DOCUHENT RESUME 



ED 315 423 



TH 014 423 



AUTHOR 
TITLE 



SPONS AGENCY 
PUB DATE 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



ABSTRACT 



Facione, Peter A. 

Critical Thinking: A Statement of Expert Consensus 
for Purposes of Educational Assessment and 
Instruction. Research Findings and 
Recommendations . 

American Philosopliical Association, Newark, Del. 
90 

112p.; Some pages have broken type. 
Reports - Research/Technical (143) 

MF01/PC05 Plus Postage. 

Cognitive Ability? *Critical Thinking; ^Curriculum 
Development; Curriculum Evaluation; "Delphi 
Technique; "Educational Assessment; Qualitative 
Research 

"Experts; *Panel Consensus Technique 



Using a qualitative research methodology, known as 
the Delphi Method, an interactive panel of experts was convened to 
work toward a consensus on the role of critical thinking (CT) in 
educational assessment and instruction. In Delphi research, experts 
participate in several rounds of questions that require thoughtful 
and detailed responses. Panelists work toward consensus by sharing 
reasoned opinions and reconsidering the opinions with regard to 
comments, objections, and arguments offered by other experts. A total 
of 46 scholars, educators, and leading figures in CT theory and CT 
assessment research were gathered for the panel meetings. About half 
of the panelists were primarily affiliated with philosophy 
departments; the others v/ere affiliated with education, social 
sciences, or physical sciences. Recommendations resulting from the 
discussion rounds address the cognitive skill dimension of CT, the 
dispositional dimension of CT, and specific recommendations on CT 
instruction and assessment, including development of a CT curriculum. 
A discussion of commercially available CT assessment tools, a 
bibliography with an emphasis on assessment, and a set of letters 
which chronicle the progress of the Delphi research group are 
appended. (TJH) 



It**************************** ****A*:^***^;*^^**^g(^^^^^^^^5^^^^^^^^^^^^j^^^ 

* Reproductions supplied by EDRS are the best that can be made 

* from the original document. 



U.S. DCPAnTMENT OF EDUCATION 

Office of Educalionai rtewarch and ImDrovemem 

eOUCATIONAURJiSOURCES INFORMATION 
CENTER (ERIC) 

P This documeni has been repfoduced as 
received from the person or orgameaiior 
ortgtnaltng it 

r Minor chances have been made to improve 
reproduction quality 

• f o»nts of v»ew or opinions stated m this docu- 
ment do not nwcessariiy represent official 
OEHI position or policy 



"PERMISSION TO REPRODUCE THIS 
MATERIAL HAS BEEN GRANTED BY 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC)." 



CRITICAL THINKING; 



A STATEMENT OF EXPERT CONSENSUS 



FOR PURPOSES OF 



EDUCATIONAL ASSESSMENT AND INSTRUCTION 



Research Findings and Recommendations 
Prepared for the 
Committee on Pre-College Philosophy 

of the 

American Phiiosophical Association 



by 

Peter A. Facione, 
California State University, Fullerton 



(c) 1990 P. A. Facione 



BEST COPY AVAILAbLL 



CONTENTS 

I — The Critical Thinking Movement and CT Assessmer;,t 1 

II •"- Research Methodology and Purpose 4 

III The Cognitive Skill Dimension of Critical Thinking 8 

IV The Dispositional Dimension of Critical Thinking 20 

Procedural, Laudatory and Normative Uses the Term "CT"..21 
Dispositions of the Good Critical Thinker 27 

V — Further Recommendations on CT Instruction and Assessment .. 27 

The CT Goal 28 

The CT Curriculum 30 

The CT Assessment Tool 30 

The CT Instructor 32 

VI — The Delphi Research Panel 33 

TABLE 1 Consensus Statement Regarding Critical Thinking 

and the Ideal Critical Thinker 3 

TABLE 2 Project History 7 

TABLE 3 Consensus List of CT Cognitive Skills and Sub-Skills .... 12 

TABLE 4 Consensus Descriptions of Core CT Skills and Sub-Skills . 13 

TABLE 5 Affective Dispositions of Critical Thinking 26 

TABLE 6 Consensus Statement on Teaching and Assessing CT Skills. 29 

TABLE 7 Participating Critical Thinking Experts 35 

APPENDICES 

Appendix A: Commercial CT Assessment Tools 36 

Appendix B: CT Bibliography with Emphasis on Assessment 40 

Appendix C: The Delphi Research Letters 51 



CRITICAL THINKING: A STATEMENT OF EXPERT CONSENSUS 
FOR PURPOSES OF EDUCATIONAL ASSESSMENT Al^D INSTRUCTION 

I ziz IhS Critical Thiriking Movement ^nci QI Assessment 

From New Jersey to California, and from Newfoundland to Florida, 
leaders in the critical thinking movement have advocated major 
educational reform. They have argued that effective and meaningful 
education requires that curricular, pedagogical and assessment strategies 
at all levels of education be coordinated so as to foster in students 
those cognitive skills and habits of inquiry associated with critical 
thinking. They have made the case that educating students to be critical 
thinkers is vital for the students themselves and for society in general, 
(Ennis, 1962, 19S1, 19B6; Passmore, 1967; Schievella, 19 iB; Sheffler, 
1973; Lipman, 1977; Siegel, 1980, 198B; Gardner, 1933; Arons, 1933; 
Beyer, 1935; Costa, 1985; Quellmalz, 1983, 1985; Scriven, 1985; 
Sternberg, 1985; Ruggiero, 1938; Paul, 1933 (a) and <b>; etc.). 

The arguments for critical thinking have been successful- 
After decades of relative neglect, the eighties witnessed a growing 
accord that the heart of education lies exactly where traditional 
advocates of a liberal education always said it was — in the processes 
of inquiry, learning and thinking rather than in the accumulation of 
disjointed skills and senescent information. The critical thinking 
movement gained momentum throughout the decade. Conferences and position 
papers led to the development of college level critical thinking (CT) 
courses. In elementary and secondary schools (K-12) teachers revised 
lesson plans to incorporate CT objectives. In the span of a few years 
publishing CT textbooks and offering CT staff development programs became 
growth industries. The CT movement enjoyed major success when 



universities introduced CT requirements into their general education 
programs and state departments of education targeted CT in tfieir 
curricular frameworks and their standardized testing programs. By the 
decade's end CT could no longer be characterized as a cottage industry. 

With success come questions: Not new ones necessarily, but, because 
of the evjpectations which have been raised and the investments being 
proposed, vexing ones. Intuitively, CT instruction should focus on how 
students approach a question and reason about it. CT pedagogy should 
develop in students those cognitive skills and affective dispositions 
which characterize the good critical thinker. Rather than or in addition 
to targeting whether a given answer is correct, CT assessment should 
target the quality of the critical thinking the students put into 
arriving at that answer. Thus, for all of their successes, CT experts 
find they must continue to address some fundamental academic concerns. 
Wh«4t exactly are those skills and dispositions which characterize CT? 
What are some effective ways to teach CT? And how can CT, particularly 
if it becomes a campus-wide, district-wide or statewide requirement, be 
assessed? 

Wh«n these academic questions are asked by the individual professor 
or teacher seeking to introduce CT into her own classroom, they are 
difficult enough. But the questions take on social, fiscal, and 
political dimensions when asked by campus curriculum committees, school 
district offices, boards of education, and the educational testing and 
publishing industries- This is not to say that the experts find these 
questions insurmountable. On the contrary, CT experts have worked with 
their colleagues in the education community on some remarkable projects. 
For example, California and New Jersey have established ways of 
introducing CT into their curricular frameworks and statewide testing 



programs. The twenty-campus California State University system* which 
enrolls hundreds o-f thousands of students* has established a process for 
the approval of CT courses for its general education requirement. 

Givon the central role played by philosophers in articulating the 
value* both individual and social* of CT* in analyzing the concept of CT* 
in designing college level academic programs in CT, and in assisting with 
efforts to introduce CT into the K-12 curriculum* it is little wonder 
that the American Philosophical Association* through its Committee on 
Pre-CoUege Philosophy, has taken an interest in the CT movement and its 
impact on the profession. In December of 1987 that committee asked this 
investigator to make a systematic inquiry into the current state of CT 
and CT assessment. 

TABLE 1 

CONSENSUS STATEMENT REGARDING CRITICAL 
THINKING AND THE IDEAL CRITICAL THINKER 

We understand critical thinking to be purposeful, self-regulatory judgment 
which results in interpretation, analysis, evaluation, and inference, as well as 
explanation of the evidential, conceptual, methodological, criteriological, or 
contextual considerations upon which that judgment is based. CT is essential as a tool 
of inquiry. As such, CT is a liberatingforce in education and a powerful resource ui one's 
personal and civic life. While not synonymous with good thinking, CT is a pervasive 
and self-rectifying human phenomenon. The ideal critical thinker is habitually 
inquisitive, well-informed, trustful of reason, open-minded, flexible, foir-minded in 
evaluation, honest in fadng personal biases, prudent in making judgments, willing to 
reconsider, clear about issues, orderiy in complex matters, diligent m seeUng relevant 
information, reasonable ui the selection of criteria, focused in inquiry, and persistent 
in seeking results which are as precise as the subject and the circumstances of inquiry 
permit. Thus, educating good critical thinkers means working toward this ideal. It 
combines developing CT skills with nurturing those dispositions which consistently 
^yield useful insights and which are the basis of a rational and democratic society. 



As Table 1 suggests, a key result of inquiry is the articulation by 
a panel of CT experts of a conceptualization of CT it terms of two 
dimensions: cognitive skills and affective dispositions. Section II of 
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this report describes the Delphi research methodology. Section III 
address the skill dimension of CT, and Section ZV the dispositional 
dimension of CT, Research findings are presented throughout the report, 
both in the text and in tabular form. Six recommendations are presented 
in Sections III and IV so they can be related most sensibly with their 
rationale. Nine additional recommendations which pertain specifically to 
CT instruction and assessment are presented in Section V- 

11 — Research Mfath odoloav and Purpose 

The Committee on Pre-College Philosophy suggested several persons 
with special expertise in CT and CT whom this investigator might contact 
as part of the inquiry into the controversial issues known to lie at the 
heart of the profession's concern. This investigator decided to employ 
the powerful qualitative research methodology known as the Delphi Method. 
The Delphi Met^iod requires the formation of an interactive panel of 
experts. These persons must be willing to share their expertise and work 
toward a consensus rf?solution of matters of opinion. Using the firs.h 
group of tjvjperts to ntsminate others, the Delphi panel soon took shapp. 
In all forty-six persons, widely recognised by their professional 
colleagues to have special experience and expertise in CT instruction, 
assessment or theory, made the commitment to participate in this Delphi 
project- If it were not for their conscienticas effort, (for which this 
inve'stigator is es'tremely appreciative;, the consensus expressed in this 
report could oot have? been reotched. 

In Delphi research experts participate in several rounds of 
questions which call for thoughtful and detailed responses. Achieving a 
consensus of expert opinion using the Delphi Method is not a matter of 
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voting or tabulating quantitative data. Rather the expert panelists work 
toward consensus by sharing their reasoned opinions and being willing to 
reconsider them in the light of the comments, objections and arguments 
o-ffered by other experts. In Delphi research, once an expert expresses 
an opinion, even a dissenting one, it becomes a -factor in the mix and 
•flow of all subsequent argument and thought. To circumvent undue 
influence arising from any given B>tpert's r- of essional status, each round 
of questions is initiated by the prciject director and all responses are 
coordinated through that person. The project director circulates to the 
entire panel direct q^ibtations and synthesized responses, with the names 
of their authprs removed. 

Thp expert panelists themselves, through the thoughtfulness and 
pisrsuasiveness of their written responses, shape the line of inquiry. 
The project director endeavors to frame questions which respond to the 
direction panel debate is taking and lead the conversation toward 
fruitful resolution. As the inquiry proceeds, the project director 
assists the panelists with bibliographies and alerts them to other useful 
sources of relevant information. As areas of accord or disagreement 
emerge these are presented to the panel in the form of drafts of 
preliminary findings or crucial follow-up questions. The process 
terminates when the project director cetermines that sufficient accord 
has been reached for areas of consensu ^ to be made public. Delphi 
findings also include descriptions of residual disagreement and 
statements of minority opinion. 

A clear and accurate conceptualization of CT is absolutely essential 
for th.e development of valid CT assessment tools and effective CT 
ifistructional programs. 'Jith this in mind, and recognizing that 
divergent conceptualizations of CT have hindered curricular and 
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assessment efforts, early in the Delphi process the panel decided its 
most worthwhile contribution could be the articulation of a clear and 
correct conceptualization of CT. The expert panelists devoted their 
major effort toward that end. The experts hoped that by coming to 
consensus they could offer educators interested in CT assessment or 
instruction a conceptualization of CT of sufficient clarity, accuracy and 
richness to warrant their serious attention. 

To balance the theoretical with the practical, the experts asked 
themselves what a generally educated college lower division level 
critical thinker should be able to do. However, they did not attempt to 
describe the typical colleQe level critical thinker. It soon became 
evident that the experts were actually articulating an ideal. It may be 
that no person is fully adept at all the skills and sub-skills the 
experts found to be central to CT. It may be that no person has fully 
cultivated all the affective dispositions which characterize a good 
critical thinker. Also humans compartmentalize their lives in ways that 
CT is more active and evident in some areas than in others. This gives 
no more reason to abandon the effort to infuse CT into the educational 
system than that knowing no friendship is perfect gives one reason to 
despair of having friends. The experts' purpose in putting the ideal 
before the education community is that it should serve as a rich and 
worthy goal guiding CT assessment and curriculum development at all 
educational levels. 
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TABLE 2 

PROJECT HISTORY 

Round 1 (Feb. 11, 1988) and Round 2 (Mar. 14, 1988) initiated the Delphi 
process. In both rounds panelists were invited to nominate other CT experts to join 
in this research project. The experts reached consensus on the working assumption that 
"the concept of CT could be made operational to the extent that important parts of CT 
could be assessed validly and reliably." The experts agreed to begin their analysis 
of CT by "identifying the core elements of CT which might reasonably be expected 
at the freshman and sophomore general education college level." The rationale for this 
decision was that the college level theoretical construct of CT could reasonably be used 
to guide what might be said about CT at the K-12 level. Also the panelists noted that 
most of the participating experts had greater experience at the college level than in K- 
12 education. 

Round 3 (Mi\y 4, 1988) was an open -ended m vitation for experts to write their own 
listoftheoperationswhichtheyconceivedofascentral toCT. The first synthesis of this 
mput was presented for expert review in Round 4 (Sept. 23, 1988). This synthesis 
focused on the skill dimension ofCT. Round 4 invited responses regarding each skill 
and sub-skill identified, a proposed [and ultimately rejected] input/output model of CT 
operations, a list of closely related cognitive operations which might or might not be 
distinguished from CT, a general statement regarding what a skiU is and how one is 
taught, and ii list of caveats and cautions regarding CT instruction and assessment. 

Round 5A (Feb. 28, 1989) reviewed the definitions and classification of CT 
cognitive skiUsin the lightofexpert responses to Round4. Round 5B (also Feb. 28, 1989) 
proposed stsitements regarding the dispositional dimension of CT and about its possible 
normativeconnotations. RoundSC(Mar. 10, 1989) asked for specific recommendations 
regarding iTT instruction and assessment, and offered a revision of the general 
statement on teaching and assessing a cog^iitive skill. Round 5 included several 
quotations culled from the panelists' earlier responses and hivited comments and 
reactions. 

The experts' comments regarding the various quotations included in each round 
added greatly to the project duector's understanding of the experts' overall views. From 
these and the responses to specific Round 5A, SB and 5C questions, the project director 
assembled a draft report of all Delphi trndin^fi, including recommendations. Round 6, 
(Sept. 25, 1989) circulated that draft and gave the CT experts the opportunity to express 
their views or make comments for inclusion in the final report, which went through its last 
revisions in Nov. 1989. 
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Ill - • The Cogpitlve Skill D lmenalon of C ri tical Thlnkli^t^ 



FINDING: As indicated in Table 1, the experts find good critical 
thinking to include both a skill dimension and a dispositional 
dimension. The experts find CT to include cognitive skills in (1) 
interpretation, (2) analysis, (3) evaluation, (4) inference, (5) 
explanation and (6) self-regulation. Each of these six is at the core 
of CT. Associated with each are criteria by which its execution can be 
meaningfully evaluated. However, no attempt is made here to specify 
those criteria since ample criteriological discussions exist in the 
1 iterature. 

Concerned not to generate misunderstandings, the ei^perts offer many 
cautions about the analysis of CT in terms of skills and sub-skills. The 
experts warn that good CT is not rote, mechanical, unreflective, 
disconnected execution of sundry cognitive processes. They caution not 
to lose sight of the whole while attempting to attend well to its many 
parts. 



RECOMMENDATION 1: All CT instruction should aim at developing good 
critical thinkers persons who can integrate successful execution of 
various skills in the CT enhanced classroom with the confidence, 
inclination and good judgment to use these powerful tools in their 
other studies and in their everyday lives. Persons who have 
proficiency in CT skills but fail to use them appropriately are most 
unlikely to be regarded as good critical thinkers. 

RECOMMENDATION 2: Those who seek to infuse CT into the educational 
system to be guided by a holistic conceptualization of what it means to 
be a ^ood critical thinker. That some aspects of CT, particularly 
features within its skill dimension, are more readily targeted by 
existing educational assessment strategies should not distort the 
conceptualization of CT nor truncate full-blown CT instruction. 



The experts characterize certain cognitive skills as central or core 
CT skills. The more one achieves proficiency in these skills, the more 
worthy one is of being regarded as adept at CT. The experts are not, 
however, saying that a person must be proficient at every skill to be 
perceived ds having CT ability. Considering the panel's purposes and 
•nethodology, trying to analyse CT in terms of necessary and sufficient 
conditions would have had strong negative utility. Thus, in view of the 
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pi ecision iMhich the question permits, the panel, early in the Delphi 
process, decided to strive for a consensus on the core skills. The panel 
was not asked to name skills without which a person is surely not a 
critical thinker. 

Responses to Rounds 4 and 5A reveal the experts to be virtually 
unani»Dous <N>95>C) on i<":luding analysis, evaluation, and inference as 
cep^ ral to CT. But in response to Round 6 one assessment expert strongly 
dissented regarding ths inclu-sion of interpretation, arguing that it was 
properly a part of comiounication, iiot CT. The same expert noted that 
analysis, as defined in this report, overlaps with reading and listening. 
These paints raise obvious difficulties for CT assessment, particularly 
as one aLtt^mpts to make finer differentiations between CT and 
communication or between analysis-in-the-CT-sense and analysis-in~the- 
rwejding-sense. Regarding self-regulation the expert said, "I think this 
is where te-iting must nterge with teaching." In response to Round 6 
anothier .i.^ises'rsm&nt e:,i5&>rt pointed out that, as comparerJ to the others, 
self -regulation .appears to be a skill of a different kind or level. In 
'rielf-regulation one applies the other CT skills to one's own CT, by, for 
ir?xample, t5valuat.\ng ore's own inferences. This gives CT an interestingly 
r^i'C'.'t '-iive character. However, as this expert noted, the meta-cognitive 
asj'ect of self-regulation makes it extremely difficult to assess using 
the standard kinds of paper and pencil instruments. Nonetheless, strong 
cons^^nsus (N 87V.) fv.ists that interpretation, e;!planation and self- 
reyulatiofi are <.£?ntr<:il Iq CT. L"For detailed results see the response 
tc^bles ..)n ft Age 10 of the Delphi letter for Round 5A in Appendix C.3 
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FINDING; There is consensus that one night improve one's own CT in 
several Hays. The experts agree that one could critically canine and 
evaluate one's own reasoning processes. One could learn how to think 
fliore objectively and logically. One could expand one's repertoire of 
those fflore specialized procedures and criteria used in dirferent areas 
of human thought and inquiry. One could increase one's base of 
information and life experience* 

It Mas readily apparent that the experts do not regard CT as a body 
of knowledge to be delivered to students as one more school subject along 
with others. The panel sees CT, like reading and writing, as having 
applications in all areas o-f life and learning. And, as several pointed 
out, CT instruction, like reading and writing, can occur in programs rich 
with discipline-specific content or in programs which rely on the events 
in everyday life as the basis for developing one's CT. 

FINDING One implication the experts draw from their analysis of CT 
skills is this; "while CT skills themselves transcend specific subjects 
or disciplines, exercising them successfully in certain contexts 
demands domain-specific knowledge, some of which may concern specific 
methods and techniques used to make reasonable judgments in those 
speci f ic contexts. " 

Although the identification and analysis of CT skills transcend, in 
significant ways, specific subjects or disciplines, learning and applying 
these skills in many contexts requires domain-specific knowledge. This 
domain-specific knowledge includes understanding methodological 
principles and competence to engage in norm-regulated practices that are 
at the core of reasoncible judgments in those specific-contexts. The 
explicit mention of "evidential, conceptual, methodological, 
cr iter iologicrfl, or contextual" considerations in connection with 
ex|«lanation reinforces this point. Too much of value is lost if CT is 
conceived of simply as a list of logical operations and domain-specific 
knowledge is conceived of simply as an aggregation of information. 
Inquiry into the nexus of reasonable judgment and actual application can 

10 
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produce new appreciations of the necessity of robust concepts of both CT 
and domain-specific knowledge in education. 

RECOMMENDATION 3i Since becoming adept at CT involves learning to use 
CT skills eHectively in many diHerent contexts, the experts insist 
that "one cannot overemphasize the value o^ a solid liberal education 
to supplement the honing of one's CT skills and the cultivating of 
one's CT dispositions." 

The experts caution that CT skills can usefully be grouped and sub- 
classified in a number of legitimate ways. Hence, the sub-classification 
which resulted from this Delphi research should not be interpreted as 
necessarily excluding all others. Indeed, while declaring themselves to 
be in agreement with this sub-classification, various participating 
experts have also published their own sub-classifications. There are 
areas of overlap in the classification system which emerged from tfie 
Delphi research. However, while characterizing each skill and sub-skill 
is important, creating arbitrary differentiations simply to force each 
and every «ub-skill to become conceptually discrete from all the others 
is neither necessary nor useful. In practical contexts the execution of 
some skills or sub-skills may presuppose others- Thus, order of the 
Delphi listing is not intended to imply the endorsement of any 
psychological, logical or epistemological order or skill-sequence, nor as 
prescribing any educational taxonomy or skill-hierarchy. 

Table 3 lists the skills and sub-skills which the experts identify 
as being at the core of CT. No claim is being made that the list 
e>{hausts the concept of CT in either breadth or detail. Beyond their 
inclusion in CT, many of the skills and sub-skills identified are 
valuable, if not vital, for other important activities, such as 
ccMfimunicating effectively. Also CT skills can be applied in concert with 
other technical or interpersonal skills to any number of specific 



concerns such as programming computers, defending clients, developing a 
winninc) sales strategy, managing an office, or helping a friend figure 
out Mhat might be wrong with his car. In part this is what the experts 
mean by characterizing these CT skills as peruasi^/^e and purposeful* It 
is also fair to say that a particular skill, such as evaluation, or a 
particular sub-skill, such as developing reasons, is essential for 
success in a given endeavor, such as properly diagnosing illness. The 
experts are not concerned that various skills and sub-skills are widely 
used. It is not a problem that the skills might be essential elements in 
other endeavors. Qn the contrary, it would be extremely disconcerting if 
they were not, since the case for infusing CT into the educational system 
depends un CT's utility across almost all areas of life and learning. 

The experts are clear on the point that not every useful cognitive 
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TABLE 3 



CONSENSUS LIST OF CRITICAL THINKING 
COGNITIVE SKILLS AND SUB-SKILLS 



1. Interpretation 



* Categorization 

* Decoding Significance 

* Clarifjfing Meaning 



2. Analysis 



* Examining Ideas 

* Identifyifig Arguments 

* Analyzing Arguments 



3. Evaluation 



* Assessing Claims 

* Assessing Arguments 



4. Inference 



* Querying Evidence 

* Conjecturing Alternatives 

* Drawing Conclusions 



5. Explanation 



* Stating Kesults 

* Justifying Procedures 

* Presenting Arguments 



6. Self-Regulation 



* Self-examination 

* Self-cofiection 
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process should be thought of as CT. Not every valuable Lhinking skill is 
CT skill. CT is one among a -family o-f closely related forms of higher- 
order thinking, along tmith, for example, problem-solving, decision 
making, and creative thinking. Unfortunately the conceptual overlaps and 
complex relationships Among all the various forms of higher-order 
thinking have yet to be examined satisfactorily. However, that does not 
imply that one cannot develop a careful and accurate conceptualisation of 
the tarcjet, CT — a conceptualization fully adequate to its purpose, 
which is to guide CT assessment and instruction. 

In addition to accord on the listings in Table 3, the Delphi experts 
find remarkable consensus on t^;e descriptions of each of the skills and 
sub—skills. These descriptions are presented in Table 4. The examples 
as'^aciated with each sub-skill are intended as clarifications. Some 
readers aiight see in them suggestions of possible instructional or 
assessment strategies. Others might see in them the tools to initiate 
staff development conversations about the curricular implications- 
However, the panel's consensus has to do with the skill and sub-skill 
descriptions, at id does not necessarily extend to the examples. 




TABLE 4 

CONSENSUS DESCRIPTIONS OF CORE CT SKILLS AND SUB-SKILLS 

1. INTERPRETATION: To comprehend and express the meaning or 
significance of a wide variety of experiences, situations, data, 
events, judgments, conventions, beliefs, rules, procedures or 
criteria. 

1.1 CATEGORIZATION: 

* to apprehend or appropriately formulate categories, 
distinctions, or frameworks for understanding, describing or 
characterizing information. 

* to describe experiences, situations, beliefs, events, 
etc. ,'io that they take on comprehensible meanings in terms of 
appropriate categorisations, distinctions, or frameworks. 

For eu'iimple! to recogniic; d problem and define its character 
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without prejudice to inquiry; to deterifline a useful way of sorting 
and sub-classifying inf ormationj to make an understa. dabl e report 
of Mhat one experienced in a given situatian; to clat^sify data, 
findings or opinions using a given classification schema. 

1.2 DECODING SIGNIFICANCE. 

* to detect, attend to, and describe the informational 
content, affective purport, directive functions, intentions, 
motives, purposes, social significance, values, views, rules, 
procedures, criteria, or inferential relationships expressed in 
convention-based communication systems, such as in language, 
social behaviors, drawings, numbers, graphs, tables, charts, signs 
and symbols. 

For example! to detect jrid describe a person's purposes in asking 
a given question; to appreciate the significance of a particular 
facial expression or gesture used in a given social situation; to 
discern the use of irony or rhetorical questions in debate; to 
interpret the data displayed or presented using a }:articular forni 
of instrumentation. 

1.3 CLARIETIN6 MEANING: 

* to paraphrase or make explicit, through stipulation, 
description, analogy or figurative expression, the contextual, 
conventional or intended meanings of words, ideas, concepts, 
statements, behaviors, drawings, numbers, signs, charts, graphs, 
symbols, rules, events or ceremonies. 

* to use stipulation, description, analogy or figurative 
expression to remove confusing, unintended vagueness or ambiguity, 
or to design a reasonable procedure for so doing. 

'or example: to restate what a person said using different words 
or expressions while preserving that person's intended meanings; 
to find an example which helps explain something to someone; to 
develop a distinction which makes clear a conceptual difference or 
removes a troublesome ambiguity. 

2. ANALYSIS: To identify the intended and actual inferential 
relationships among statements, questions, concepts, descriptions 
or other forms of representation intended to express beliefs, 
judgments, experiences, reasons, information, or opinions. 

2.1 EXAMINING IDEAS: 

* to determine the role various expressions play or are 
intended to play in the context of argument, reasoning or 
persuasion. 

* to define terms. 

* to compare or contrast ideas, concepts, or statements. 

* to identify issues or problems and determine their 
component parts, and also to identify the conceptual relationships 
of those parts to each other and to the whole. 

For example: to identify a phrase intended to trigger a 
sympathetic emotional response which might induce an audience to 
agree with an opinion; to examine closely related proposals 
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regarding a given problem ani to determine their points o^ 
similarity and divergence; given a complicated assignment, to 
determine how it might be broken up into smaller, more manageable 
tasks; to define an abstract concept. 

2.2 DETECTING ARGUMENTS: 

* given a set of statements, descriptions, questions or 
graphic representations, to determine whether or not the set 
expresses, or is intended to express, a reason or reasons in 
support of or contesting some claim, opinion or point of view. 

For example, given a paragraph, determine whether a standard 
reading of that paragraph in the context of how and where it is 
published, would suggest that it presents a claim as well as a 
reason or reasons in support of that claim; given a passage from a 
newspaper editorial, detprmine if the author of that passage 
intended it as an expression of reasons for or against a given 
claim or opinion; given a commercial announcement, identify any 
claims being advanced, along with the reasons presented in their 
support « 

2.3 ANALYZING ARGUMENTS: 

* given the expression of a reason or reasons intended 
to support or contest some claim, opinion or point of view, to 
identify and differentiate: (a) the intended main conclusion, (b) 
the premises and reasons advanced in support of the main 
conclusion, (c) further premises and reasons advanced as backup or 
support for those premises and reasons intended as supporting the 
main conclusion, (d) additional unexpressed elements of that 
reasoning, such as intermediary conclusions, unstated assumptions 
or presuppositions, (e) the overall structure of the argument or 
intended chain of reasoning, and (f ) any items contained in the 
body of expressions being examined which are not intended to be 
taken as part of the reasoning being expressed or its intended 
background. 

For example: given a brief argument, paragraph-sized argument, or 
a position paper on a controversial social issue, to identify the 
author's chief claim, the reasons and premises the author advances 
on behalf of that claim, the background information used to 
support those reasons or premises, and crucial assumptions 
implicit in the author's reasoning; given several reasons or 
chains of reasons in support of a particular claim, to develop a 
graphic representation which usefully char^icter j z es the 
inferential flow of that reasoning. 

3. EVALUATION: To assess the credibility of statements or other 
representations which are accounts or descriptions of a person's 
perception, experience, situation. Judgment, belief, or opinion; 
and to assess the logical strength of the actual or intend 
inferential relationships among statements, descriptions, 
questions or other forms of representation. 

3.1 ASSESSING CLAIMS: 

* to recognize the factors relevant to assessing the 
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degree of credibility to ascribe to d source of information or 
opinion. 

* to assess the contextual relevance of questions, 
information, principles, rules or procedural directions. 

* to assess the acceptability, the level o. confidence 
to place in the probability or truth of any given representation 
of an experience, situation, Judgment, belief or opinion. 

For example; to recognise the factors which make a person a 
credible witness regarding a given event or credible authority on 
a given topic; to determine if a given principle of conduct is 
applicable to deciding what to do in a given situation} to 
determine if a given claim is likely to be true or false based on 
what one knows or can reasonably find outt 

3.2 ASSESSING ARGUMENTS: 

* to judge whether the assumed acceptability of the 
premises of a given argument justify one's accepting as true 
(deductively certain), or very probably true (inductively 
justified), the expressed conclusion of that argument. 

* to anticipate or to raise questions or objections, and 
to assess whether these point to significant weakness in the 
argument being evaluated. 

* to determine whether an argument relies on false or 
doubtful assumptions or presuppositions and then to determine how 
crucially these affect its strength. 

* to judge between reasonable and fallacious inferences; 

* to Judge the probative strength of an argument's 
premises and assumptions with a view toward determining the 
acceptability of the argument. 

* to determine and judge the probative strength of an 
argument's intended or unintended consequences with a view toward 
judging the acceptability of the argument; 

5t to determine the extent to which possible additional 
information might strengthen or weaken an argument. 

For example: given an argument to judge if its conclusion follows 
either with certainty or with a high level of confidence from its 
premises; to chpck for identifiable formal and informal falUciei; 
given an objection to an argument to evaluate the i:;;c.ril force of 
that objection; tc evaluate tne quaii'.', ■.■inj applicability of 
analogical argumetili.; • r: judge the logiC?! strength of cirgumsnts 
based on ■■vocth«t i cdl situations cr causal rea-ionirtg; to judge if 
a given argument i ? relevant or applicable or has i mp 1 : f at i ons for 
the situation dt t;and; to determine how possible new data miL)h^ 
lead logically to the further confirmation or d i sconf i r aid t i on of a 
given :piniGr:. 

4: INFERENCE: To identify and secure elements needed to draw 
reasonable conclusions; to form conjectares and hypotheses; to 
consider relevant information and to educe the consequences 
flowing from data, statements, principles, evidence, judgments, 
beliefs, opinions, concepts, descriptions, questions, or other 
forms of representation. 
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4.1 QUERYING EVIDENCE: 

* in particular, to recognize premises which require 
support and to formulate a strategy for seeking and gathering 
information which might supply that support. 

* in general, to judge that information relevant to 
deciding the acceptability, plausibility or relative merits of a 
given alternative, question, issue, theory, hypothesis, or 
statement is required, and to determine plausible investigatory 
strategies for acquiring that information. 

For examples when attempting to develop a persuasive argument in 
support oi one's opinion, to judge what background information it 
Mould be useful to have and to develop a plan which will yield a 
clear answer as to whether or not such information is available; 
after judging that certain missing information would be germane in 
determining if a given opinion is more or less reasonable than a 
competing opinion, to plan a search which will reveal if that 
information is available. 

4.2 CONJECTURING ALTERNATIVES. 

* to formulate multiple alternatives for resolving a 
problem, to postulate a series of suppositions regarding a 
question, to project alternative hypotheses regarding an event, to 
develop a variety of different plans to achieve some goal. 

* to draw out presuppositions and project the range of 
po&sible consequences of decisions, positions, policies, theories, 
or beliefs. 

For example: given a problem with technical, ethical or budgetary 
ramifications, to develop a set of options for addressing and 
resolving that problem; given a set of priorities with which one 
may or may not agree, to project the difficulties and the benefits 
which are likely to result if those priorities are adopted in 
decision making, 

4.3 DRAWING CONCLUSIONS: 

* to apply appropriate modes of inference in determining 
what position, opinion or point of view one should take on a given 
matter or issue. 

* given a set of statements, descriptions, questions or 
other forms of representation, to educe, with the proper level of 
logical strength, thej.r inferential relationships and the 
consequences or the presuppositions which they support, warrant, 
imply or entail. 

* to employ successfully various sub-species of 
reasoning, as for example to reason analogically, arithmetically, 
dialectically , scientifically, etc. 

* to determine which of several possible conclusions is 
most strongly warranted or supported by the evidence at hand, or 
which should be rejected or regarded as less plausible by the 
information given. 

For example; to carry out experiments and to apply appropriate 
stjitistical inference techniques in order to confirm or disconfirm 
an empirical hypothesis; given a controversial issue to examine 
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in'forffled opinions, consider various opposing views and the reasons 
advanced for them, gather relevant information, and formulate 
one's own considered opinion regarding that issue; to deduce a 
theorem frotn axioms using prescribed rules of inference. 



5: EXPLARATION: To state the results of one's reasoning; to 
justifj' that reasoning in terms of the evidential, conceptual, 
methodological, criteriological and contextual considerations upon 
which one's results were based; and to present one's reasoning in 
the form of cogent arguments. 

5.1 STATING RESULTS: 

* to produce accurate statements, descriptions or 
representations of the results of one's reasoning activities so as 
to analyze, evaluate, infer from, or monitor those results. 



For exaniple: to state one's reasons for holding a given view; to 
write down for one's own future use one's current thinking about 
an iiBportant or complex matters to state one's research findings; 
to convey one's analysis and judgment regarding a work of art; to 
state one's considered opinion on a matter of practical urgency. 



5.2 JUSTIFYING PROCEDURES: 

* to present the evidential, conceptual, methodological, 
criteriological and contextual considerations which one used in 
forming one's interpretations, analyses, evaluation or inferences, 
so that one might accurately record, evaluate, describe or justify 
those processes to one's self or to others, or so as to remedy 
perceived deficiencies in the general way one executes those 
processes. 



For example: to keep a log of the steps followed in working 
through a long or difficult problem or scientific procedure; to 
explain one's choice of a particular statistical test for purposes 
of data analysis; to state the standards one used in evaluating a 
piece of literature; to explain how one understands a key concept 
when conceptual clarity is crucial for further progress on a given 
probletn; to show that the prerequisites for the use of a given 
technical methodology have been satisfied; to report the strategy 
used in attempting to make a decision in a reasonable way; to 
design a graphic display which represents the quantitative or 
spatial infornation used as evidence. 



5.3 PRESENTING ARGUMENTS: 

* to give reasons for accepting some claim. 

* to meet objections to the method, conceptualizations, 
evidence, criteria or contextual appropriateness of inferential, 
analytical or evaluative judgments. 



For example! to write a paper in which one argues for a given 
position or policy; to anticipate and to respond to reasonable 
criticisms one might expect to be raised against one's political 
views; to identify and express evidence and counter-evidence 
intended as a dialectical contribution to one's own or another 
person's thinking on a matter of deep personal concern. 
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6: SELF-REGDLATION: Self-consciously to monitor one's cognitive 
activities, the elements used in those activities, and the results 
educed, particularly by applying skills in analysis and evaluation 
to one's own inferential Judgments with a view toward questioning, 
confirming, validating, or corrocting either one's reasoning or 
one's results. 

6.1 SELF-EXAMINATION: 

* to reflect on one's own reasoning and verify both the 
results produced and the correct application and execution of the 
cognitive skills involved. 

* to make an objective and thoughtful meta-cognitive 
self -assessment of one's opinions and reasons for holding them. 

* to judge the extent to which one's thinking is 
influenced by deficiencies in one's knowledge, or by stereotypes, 
prejudices, emotions or any other factors which constrain one's 
objectivity or rationality. 

* to reflect on one's motivations, values, atuitudes and 
interests with a view toward determining that one has endeavored 
to be unbiased, fair-minded, thorough, objective, respectful of 
the truth, reasonable, and rational in coming to one's analyses, 
interpretations, evaluations, inferences, or expressions. 

For example: to examine one's views on a controversial issue with 
sensitivity to the possible influences of one's personal bias or 
self-interest; to review one's methodology or calculations with a 
view to detecting mistaken applications or inadvertent errors; to 
reread sources to assure that one has not overlooked important 
information; to identify and review the acceptability of the 
facts, opinions or assumptions one relied on in coming to a given 
point of view; to identify and review one's reasons and reasoning 
processes in coming to a given conclusion, 

6.2 SELF-CORRECTION: 

* where self-examination reveals errors or deficiencies, 
to design reasonable procedures to remedy or correct, if possible, 
those mistakes and their causes. 

Far example: given a methodological mistake or factual deficiency 
in one's work, to revise that work so as to correct the problem 
and then to determine if the revisions warrant changes in any 
position, findings, or opinions based thereon. 
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IV -- The Dispositional Dimenglon of Critical Thinking 



As is evident, particularly in the descriptions of self-examination 
and self-correction, there are dispositional components to critical 
thinking. Indeed each cognitive skill, if it is to be exercised 
appropriately, can be correlated with the cognitive disposition to do so. 
In each case a person who is proficient in a given skill can be said to 
have the aptitude to execute that skill, even if at a given moment the 
person is not using the skill. But there was a great deal more many 
experts wished say in regard to the personal traits, habits of mind, 
attitudes or affective dispositions which seem to characterize good 
critical thinkers. 



FINDING: Although the language here is metaphorical, one would find 
the panelists to be in general accord with the view that thsre is a 
critical spirit, a probing inqui siti venesB, a keenness of mind, a 
zealous dedication to reason, and a hunger or eagerness for reliable 
information which good critical thinkers possess but weak critical 
thinkers do not seem to have. As water strengthens a thirsty plant, 
the affective dispositions are necessary for the CT skills identified 
to take root and to flourish in students. 

RECOMMENDATION 4: Modeling that critical spirit, awakening and 
nurturing those attitudes in students, exciting those inclinations and 
attempting to determine objectively if they have become genuinely 
integrated with the high quality execution of CT skills are, for the 
majority of panelists, important instructional goals and legitimate 
targets for educational assessment. However, ^he experts harbor no 
illusions about the ease of designing appr opr i c>te instructional 
programs or assessment tools. 



Procedural, Laudatory and Normatii^e Uses of the Term "CT" 
The experts have a consensus regarding the list of affective 
dispositions which characterize good critical thinkers. This consensus is 
expressed in Table 5. However, whether or not these affective dispositions 
are part of the meaning of "CT" in the way that the cognitive skills are, 
was an issue which divided the experts from the first. It became evident 
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that various experts mean di-f-ferent things when they used the term "CT" in 
re-ference to its possible dispositional components. 

The deepest division is between the nearly two-thirds majority who 
hold that the term "CT" includes in its meaning a reference to certain 
affective dispositions and the roughly one-third minority who hold that 
"CT" refers only to cognitive skills and dispositions, but not to affective 
dispositions. The project director put this issue to the panel in several 
different ways, sometimes directly and at other times more obliquely. 
Responses, comments and arguments were shared, as were the objections and 
counter-arguments which they engendered. In the end the panel remained 
divided both numerically and in depth of feeling, with opposing positions 
b*3Coming more strident and entrenched as the debate continued. 

In Round SB, of those expressing an opinion, the majority (61%) 
maintain that the affective dispositions constitute part of the meaning of 

' "CT." They argue that tJiese dispositions flow from, and are implied by, 

the very cor^cept of CT, much as the cognitive dispositions are. These 
experts argue that being adept at CT skills but habitually not using them 
appropriately disqualifies one from being called a critical thinker at all. 
T^»U3, in addition to using "CT" in its procedural sense, these panelists 

1 alsu use "CT" in its laudatory sense. They find it sensible to say, "This 

person is a critical thinker, but this other person is so mentally lazy, 
close-minded, unwilling to check the facts and unmoved by reasonable 
arguments that we simply cannot call him a critical thinker," 

The laudatory use of "CT" can suggest approval of how well a person 
applies hier CT skills or it can convey praise for the person because the 
perscm has the proper effective dispositions. While the two-thirds 
mr^jority was eloquent regarding the importance of finding ways to instill 
cjffective dispositions in students, in the final analysis they were unable 
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to persuade the other third o-f their expert colleagues to view these 
dispositions as essential to the concept of CT. Tlie majority was, hoMever, 
persuasive in bringing about virtual unanimity regarding using the 
affective dispositions to describe the paradigm critical thinker, <See 
Table 1.) 

In Round SB a minority (30"/.) insist on using "CT" in a strict 
procedural sense, that is as referring only to a certain judgmental 
process. They distinguish sharply between what is true of critical 
thzfiking from what is true of good critical thinkers. Their primary 
concern is with the CT skills. They argue that good critical thinkers are 
people who have those skills and certain valuable habits as well. If they 
are good critical thinkers, then they use their CT skills appropriately 
because good critical thinkers also have some or all of the affective 
dispositions listed in Table 5. But those dispositions are not what is 
meant by "CT." They argue that one would not want to say a sophist is not 
a critical thinker simply btfcause the sophist uses CT skills for deceptive 
or self-interested ends. The sophist, they would maintain, is a critical 
thinner — but not an good one <in an ethical sense). The strict 
proceduralists do not find it sensible deny that a person is a critical 
thinker simply because the person, while skilled in CT, fails to check the 
credibility uf --iources, gives up too soon when asked to work a challenging 
problem, lacks confidence in using reason to approach everyday problems, or 
ijnores painful facts. These experts hold that such a person, because of 
his CT skills, sihould be called a critical thinker — but not a good one, 
(Ui terms of his eff active use of those skills). 

As "r-uggestwd above, there are two senses of the term "good" which 
might be operating wfien one uses the phrase "good critical thinker." One 
sense applies to the thinker's effectiveness and responds to the question, 
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"How well is this person using CT?" The second sense applies to the 
thinker's morality and responds ta the question, "Is this person's use of 
CT ethical?" In order to clari-fy which sense of "good" the experts wished 
to convey, Round 4 asked the panel to respond to a proposal that CT might 
have a normative dimension in addition to a skill dimension and a affective 
dispositional dimension. 

FINDING: The mistaken notion that CT has a normative component is 
rejected by the expert panelists. It is an inappropriate use of the 
term to deny that someone is engaged in CT on the grounds that one 
disapproves ethically of what the person is doing. What "CT" means, 
why it is of value, and the ethics o-f its use are best regarded as 
three distinct concerns. 

The majority o-f experts (52/i) forcefully reject the proposed 
normatiue use o-f "CT." They hold that it is one thing to say what 
something is, and another thing to say how it ought to be used. A 
person's skills and attitudes are what they are, even if the person 
suffers from certain ethical inadequacies. 

Only a small group <17X>, argue in favor of using "CT" in a 
normative sense. This minority of experts, all of whom also use "CT" in 
its commonly understood laudatory sense, hold that the true meaning of 
"CT" extends to a certain set of ethical norms and social values. For 
example, they would be willing to say that a defense attorney who uses CT 
skills to cause a mistrial or win acquittal for a guilty client ought not 
be dignified with the title of critical thinker. By the same token, the 
prosecutor who uses GT skills to contrive a way to mislead a gullible 
juf y into convicting and punishing an innocent person is not a critical 
thipier. Sifice neither sufficiently value truth <3,nd since both appear to 
lack the moral fiber to eftchew deliberate deception in the practice of 
their socially important professions, neither attorney should be accorded 
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the moral approval which calling them critical thinkers would imply. 

The debate turned out to be instructive in another way as well. The 
consensus (74V. in support and 4X opposed) was that this report should 
express the experts' fullest support and appreciation of the immense 
personal and social importance of CT. 

The panel shares a solid consensus about the importance of CT as a 
tool of inquiry, as a liberating force in education, as a powerful 
resource in one's personal life, and as a vital component in a rational 
democratic society. It is extremely unlikely that any panelist would 
condone using CT for immoral, deceptive, or unjust purposes. However the 
personal and civic value of CT and sensitivity to the morality of its use 
are not acceptable grounds far building a normative dimension into the 
♦neaning of the term "CT". Some even saw such an effort as misguided and 
potentially destructive of the CT movement. Giving "CT" a normative 
twist could, they argue, lead to unwarranted limitations on open inquiry 
and to unjustifiable ideological restrictions on the very concept of 
being a "thinking" person. The totalitarian specter this conjures up is 
the antithesis of the liberating critical spirit described earlier. 

Disp(i'sitiorts of the Good Critical Thinker 

FINDING: To the experts, a good critical thinker, the paradigm case, 
is habitually disposed to engage in, and to encourage others to engage 
in, critical judgment. She is able to make such judgments in a wide 
range of contexts and for a wide variety of purposes- Although perhaps 
not always uppermost in mind, the rational justification for 
cultivating those affective dispositions which characterize the 
paradigm critical thinker are soundly grounded in CT's personal and 
civic value. CT is known to contribute to the fair-minded analysis and 
resolution of questions. CT is a powerful tool in the search for 
knowledge. CT can help people overcome the blind, sophistic, or 
irrational defense of intellectually defective or biased opinions. CT 
promotes rational autonomy, intel 1 ectual freedom and the ob jecti ve, 
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reasoned and evidence-based investigation ai a very Hide range of 
personal and social issues and concerns. 

Thusi in addition to possessing CT skills the good critical thinker 
can be characterized by certain af-fective dispositions or habits of mind. 
These dispositions, listed in Table S below, -floM from two sources: 
characteristics which the experts judge to hold true of good critical 
Uunkers, and the a-Ffective dispositions the experts judge to be part of 
CT in its fullest realization. The majority of the experts (61/1) regard 
the dispositions listed in Table 5 as part of the conceptualization of 
CT. A consensus exists (83*/. in favor) that good critical thinkers can be 
characterized as exhibiting these dispositions. 

r \ 

TABLE 5 

AFFECTIVE DISPOSITIONS OF CRITICAL THINKING 

APPROACHES TO LIFE AND LIVING IN GENERAL: 
^ inquisitivenesi with regafd to a wide range of issues, 

* concern to become and remain generally well-informed, 

* alertness to opportunities to use CT, 

* trust in the processes of reasoned inquiry, 

* self-confidence m one's own abUity to reason, 

* open-nundedness reg^uding divergent world views, 

* flexibility in considering alternatives and opinions, 

* understandmg of the opinions of other people, 

* fair-mindedness m appraising reasoning, 

* honesty in facing one's own biases, prejudices, 

stereotypes, egocentric or sodocentric tendencies, 

* prudence in suspending, making or altering judgments, 

* willingness to reconsider and revise views where honest 

reflection suggests that change is warranted. 

APPROACHES TO SPBaPIC ISSUES, QUESTIONS OR PROBLEMS: 

* clarity in stating the question or concern, 

* orderliness in working with complexity, 

* diligence in seeUng relevant information, 

* ressonableness in selecting and applying criteria, 

* care m focusing attention on the concern at hand, 

* persistence though difficulties are encountered, 

* precision t o the degye e permitted by subject and circumstances. j 
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The experts are not saying that a person whose metaphysical, 
epistemological, political, cultural or religious view of the world is 
different from one's own is, ipso facto, not a good critical thinker. 
Beliefs are not atoms each of which is at any moment subject to being 
reconsidered independently. Beliefs form intricately interconnected 
systems of thought. To focus critical attention on any of them, 
particularly those more central or fundamental to one's own view of the 
world, cafi cause reverberations throughout one's entire belief system. 
Thus in advocating CT the panelists are not urging ideological 
conformity. Indeed, just as many experts argiied that an over-emphasis on 
the values of CT could lead to trouble, others warn that an over-emphasis 
on the skills dimension of CT to the exclusion of the affective 
di«5po5itions might have the unfortunate result of making some students 
close-minded, intellectually inflB>{ible and dogmatic. 

RECOMMENDATION 5: Just as with the cognitive difnension CT, when 
conceiving of the education or assessinent of critical thinkers^ it is 
important to consider ways of developing materials, pedagogies, and 
assessment tools that are effective and equitable in their focus on 
these affective dispositions. The cultivation of these dispositions is 
particularly important to insure the use of CT skills outside the 
narrow instructional setting. Persons who have developed these 
affective dispositions are much more likely to apply their CT skills 
appropriately in both their personal life and their civic life than are 
those who have mastered the skills but are not disposed to use them. 

As with the listing of cognitive sJ-ills earlier, the panel does not 
intend that each disposition be considered a necessary condition. The 
experts are characterizing the ideal. In setting forth the concept cA 
the paradigm critical thinker, they intend to express a goal toward which 
all might strive. These virtues require a measure of maturity and 
personal development not commonly -Found in college sophomores or twelfth 
graders. Yet to delc<y embarking en the practices and disciplines which 
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will lead to these vir tues would be an even more profound mistake. 



RECOMMENDATION 6: From early childhood people should be taught, ^or 
example, to reason, to seek relevant facts, to consider options, and to 
understand the views of others. It is neither impractical nor 
unreasonable to demand that the educational system teach young people 
the habits of mind which characterize the good critical thinker, 
reinforce those practices, and move students well down the path toward 
their attainment. 



V — Further Recommendations on CT Instruction and Assessment 

Several pedagogical and assessment implications folloM from the 
dispositional dimension of CT, implications which might not be apparent 
if educators focused only on the skill dimension of CT. The education of 
good critical thinkers is more than training students to execute a set of 
cognitive skills. For example, in terms of pedagogy, modeling how to 
evaluate critically that information which students would normally accept 
uncritically and encouraging them to do the same can do wonders for 
developing their confidence in their CT ability. With this confidence 
«itudents are much more likely to try thinking for themselves. Just as 
ifistruction should not focus on skills only, assessment which focus on 
skills only may give a misleading or incomplete picture of someone's 
strengths as a critical thinker. 



The CT Gasl 



RECOMMENDATION 7j Brcause CT helps students with a wide range of 
educational, personal and civic concerns in a rational way, the 
academic goal of CT instruction, regardless of the educational level, 
should be furthering students in the development of their CT cognitive 
skills and affective dispositions. 
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TABLE 6 

CONSENSUS STATEMENT ON TEACHING AND ASSESSING CT SiaiXS 



A CT skill, like any skill, is the ability to engage in an activity, process or 
procedure. In general, having a skill includes being able to do the riglit thing at the 
rigjhttime. So, being skilled at CT involves knowing, perhaps implicitly or without the 
ability to articulate this knowledge, both a set of procedures and when to apply those 
procedures. Being skilled also Involves having some degiree of proficiency in executing 
those procedures and being willing to do so when appropriate. Reflecting on and 
improving one's CT skills involves judgMig when one is or is not performing well, or as 
well as possible, and considering ways of improving one's performance. Learning CT 
involves acquiring the ability to make such self-reflective Judgpnents. 

Skills, particularly CT cognitive skills, can be taught hi a variety of ways, such 
as by making the procedures explicit, describing how they are to be applied and 
executed, explaining and modeling their correct use, and justifying their application. 
Teaching cognitive skills also involves exposing learners to situations where there are 
good reasons to exercise the desired procedures, judgjuig their performance, and 
providing the leacnerswith constructive feedback regjardingboth their profidency and 
ways to improve it. Instruction mig)lit start with situations that are artificially simple, 
but should cufaninate in situations that are realistically complex. Particularly ui the case 
of CT, the learners must contribute a solid measure of personal effort, attention, practice, 
desire, and, as they learn how, self-monitoring. Teaching skills involves motivating 
learners to achieve higher levels of proficiency and, particularly in the case of CT, 
independence. It also involves coaching learners on how they can achieve those goals. 

In theory there are several ways persons can be judged to t>e more or less 
proficient in a given CT skill or at the integrated use of related CT skills. One way 
is to observe a person over time performmg those activities, processes or procedures 
generally regarded as presupposing that skill for proper execution. One then mak«..* < 
judgnient regarding the degree to which tlie person possesses the general skill in 
question. A second way is to compare the outcomes (if any) that result horn executing 
a given skill against some set of criteria. A third way is to query persons and receive 
their descriptions of the procedures and judgments they are using as they exercise tliat 
skiU, would use if they were to perform that skiU, or did use when they performed that 
skUl. A fourth way is to compare the outcomes (if any) that result from performing 
another task against some set of criteria, where the performance of that other task 
has been shown to correlate strongly with exercising the skill of interest. However, 
that such correlations exist between any other task and CT, or any of its sut)-skills, has 
yet to be established m the research literature. 

Each of the four ways of CT assessment has limitations as well as strengths. No 
matter which ways are used, it is important to ensure that the assessment conditions 
foster an attitude in which the subjects are disposed to use their skills as well as they 
can, and are not constrained or inhibited from doing so. In our view it is highly 
advantageous to gather evidence regardmg CT performance in many situations, using 
several assessment methods, so as to compile a composite picture of the subject and to 
^ cross check the results of any one way of assessment. 
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Either to transform CT into one subject field among others, or to 
narrow the range of CT applications strictly to domain-specific subject 
content, Mould be to truncate its utility, misapprehend its nature and 
diminish its value?. Within the overall curriculum the goal of learning 
CT can be clearly distinguished from the goal of learning domain-specific 
content. And yet., while these two goals can be* distinguished, the 
experts do not wish to deny one of the best ways to learn CT is within a 
subject context. 



RECOMMENDATION 8: Direct instruction in CT and assessment oi CT should 
be an explicit parts oi any course granted approval -for purposes oi 
satisfying CT requirements, whether that course is a CT course per se 
or a course in a given subject field. The primary acadeaic criterion 
in the evaluation of a proposed instructional program for purposes of 
achieving the CT goal should be whether the program will further the 
development of students' CT skills and dispositions. 

The CT Curriculum 

Given that CT has, in many cases, became a college general education 
requ'.irement, secondary schools can be expected to begin to develop 
college preparatory CT programs. However, the value of CT extends well 
beyond its importance as a university-level inquiry tool. CT is vitally 
important in the personal and civic life of all members of society- A 
significant percentage of the citizenry will not graduate from high 
school, or if they graduate, will not have the benefit of post-secondary 
education. 



RECOMMENDATION 9: Thus, CT instruction should not be reserved only for 
those who plan to attend college. Nor should it be deferred until 
college, since it is not likely to be effective if it were. 

RECOMMENDATION 10: Explicit attention to the fostering of CT skills and 
dispositions should be made an instructional goal at all levels of the 
K-12 curriculum. The cultivation of CT dispositions and an insistence 
on giving and evaluating reasons, should be an integral part of 
elementary school education. In middle schools and high schools, 
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instruction on various aspects and applications oi CT should be 
integrated into all subject area instruction. Specific courses in CT 
and an advanced placeinent examination program in CT for college bound 
students should be developed. Although for good reasons at the post- 
secondary level CT programs are generally associated with departments 
of philosophy, no academic unit should be restricted in principle from 
participating in an institution's CT programi provided that the overall 
institutional program in CT equips students to apply CT to a broad 
range of educational, personal and civic subjects, issues and problems. 



There is growing evidence of the successes, both scientific and 
economic, of those industrialized democracies which emphasize demanding 
academic assessment and f^et firm educational standards for career and 
professional advancement. Assessment that counts is unquestionably a key 
factor in promoting academic achievement. 



RECOMMENDATION 11: Thusi fliininium CT proficiency expectations should be 
set for each educational level, including promotion in grade, high 
school graduation, college entrance, and graduate school admission. 



The CT Assessaent 
The development of valid and reliable assessment strategies from 
which teachers can draw reasonable inferences about students' CT, in 
contrast to their domain-specific knowledge or other academic abilities 
(such as reading or writing), is essential. CT assessment strategies, 
whether for use in the individual classroom or for broader purposes, must 
i^ot simply reward arriving at correct answers. They must, however 
recognize achieving correct answers by way of good CT. The challenge of 
CT assessment is not to let what is easily measured restrict our sense of 
the fullness of CT. It would be shameful if those assessment instruments 
which focus onl/ on CT skills drove our CT curricular design and caused 
the disposition £\1 ccnponents of good CT to be neglected. 

RECOMMENDATION 12: In evaluating the acceptability a CT assessment 
stratpqy or in?trument one should consider content validity^ construct 
validity, reliabilityi and fairness. 
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<l) Content V/alidity: The strategy or instrument should be based 
on an appropriate conceptualisation o-f CT and a clear understanding of 
which aspects o-f CT the assessment targets. Each task or question should 
be evaluated to insure that correctly responding to that item is not a 
matter o-f rote learning or in-Formation recall. Whether -for the classroom 
or -for broader educational purposes, CT assessment should include 
strategies -for targeting CT's dispositional dimension as well as its 
cognitive skills dimension. 

(2> Construct Validity: In acceptable CT assessment each task or 
question should have been evaluated to insure that students who answer 
correctly do so on the basis o-f good CT and that inadequate or wrong 
responses are the result o-f weak or inadequate CT. Entire strategies or 
Bpeci-fic items on which good CT leads to wrong answers, or poor CT to 
right answers, should not be used. 

<3) Reliability: In acceptable CT assessment each task or question 
should have been evaluated to insure that good critical thinkers 
generally do better on that item than weak critical thinkers. If 
different persons are involved in evaluating the results, for example 
grading essays or judging presentations, the evaluations of the different 
judges should be cross-checked to assure that their findings are 
reliable, that is, generally consistent with one another. However, it is 
an open question whether the levels of achievement associated with the 
different CT sub-skills and affective dispositions are positively 
correlated. Empirical research on how the sub-skills correlate with each 
otfier and with various dispositions has yet to be undertaken. Thus, at 
this time, due caution should be exercised regarding how to interpret 
t«?chnical aiea-iur.^s uf test-form reliability in the case of paper and 



pencil CT 4assessfnent instrumsnts. 

<4) Fairness: CT assessment should not unfairly disadvantage or 
advantage groups of students on the grounds of reading ability, domain- 
specific knowledge [broadly understood as including the evidential, 
conceptual, methodological, criteriological, contextual considerations, 
or familiarity with technical vocabulary!!, gender or age related life 
experience, ethnicity or socio-economic status, differences in social 
norms, or differences in cultural assumptions. CT assessment locates CT 
tasks and questions in some assumed context, either subject -specific, 
everyday life, or fictional. Thus, guaranteeing that all students, 
regardless of their individual backgrounds, will come to the CT 
assessment on a perfectly equal basis in terms background knowledge, 
reading ability, life experiences, etc. is impossible. However, 
examining 5:he assessment strategy or instrument to be sure that these 
factors do not UTffairly influence the results is prudent and reasonable. 
Although one cannot eliminate the influence of these variables, one may 
be c«ble to neutralise or control for their affects. 

The fairnesjs criterion applies both to discipline-neutral and 
difscipline-specific CT assessment. Within curricular programs 
discipline-specific CT assessment is encouraged, since it is possible for 
one to be fair in one's presumptions regarding subject-specific criteria, 
concepts, methodologies, evidence, information and terminology. The 
challenge of such assessment is to factor out the discipline content in 
order to access the strength or weakness of the CT. It is worti-^ noting 
*:hat discipline-neutral CT assessment also makes similar assumptions 
regarding the everyday contexts which form its topic content. 

RECOMMENDATION 13: CT assessment should occur frequently, and it should 
be used di agnosti cal 1 y as well as summati vely . DiHerent kinds ai 



instruments should be etnployed, depending on which aspect oi CT is 
being targeted and where students are in their learning — the 
introductory stage, the practice stage, the integration stage or the 
generalized transfer stage. Although the veteran CT instructor is ablG 
to assess students continuously, CT assessment should be made explicit 
to reinforce its worth in the eyes oi the students, their families, and 
the public. It should be made explicit to support the goals of 
educators seeking to improve the curriculum. And it should be made 
explicit to properly inform educational policy formation. 



The CT Instructor 

RECOMMENDATION 14: Teaching CT is most effective if the instructor 
models CT dispositions and the proper use of CT skills in the very 
process of instruction. Regardless of the subject area, students 
should be encouraged to be curious, to raise objections, ask questions, 
point out difficulties in the instructor's position. These objections 
and questions should be clarified, interpreted, and examined 
objectively. Students should be given reasons for doing things a 
certain way, rather than being dogmatically told how to do them. 
Instruction should bridge the gap between the subject and the student's 
own experience. In the case of CT instruction, the topics of 
discussion should not be restricted to factual matters or academic 
subjects, but shtjuld include issues which have normative, moral, 
ethical or public policy dimensions. 



The ideal CT instructor will integrate instruction in CT in a 
variety o-f subject areas. She will teach speci-fic CT skills directly 
using these subjects as content for the application of those skills- She 
will help students elaborate, transfer and generalise these skills to a 
variety o-f contexts. She will create a classroom and school environment 
which is supportive of CT. She will model CT in her teaching and her 
interactions with colleagues. She will provide her students with 
thought-provoking subjects to learn about, and projects to undertake. 
She will engage students in social activities requiring them to reflect 
on, articulate, share and di<»cuss justifications, explanations and 
contrasts in how tl>ey executed various CT tasks. She will evaluate each 
student's* progress, achievement or proficiency in CT continuously. 

RECOMMENDATION 15: For CT to infuse the K-12 and college curr i cul um , 
teacher "training" should give way to teacher "education." If teachers 



&re to model CT, so must those who have an instructional role in 
teacher preparation or staf^ development. In all instruction, and 
particularly in CT instruction, both faculty and leaders of faculty 
development should model CT. They should foster the students' 
confidence in their own powers of reason, rather than dependency on 
rote learning. They should nurture in students open-mi ndedness, 
attention to alternatives, and as much precision of thought as the 
subject and circumstances permit. 

VI — The Delphi Research Panel 

The Delphi research findings reported here result from the 
participation of forty-six scholars, educators and leading figures in CT 
theory and CI assessment research. Roughly half the panelists are 
primarily affiliated academically with Philosophy <52%), the others are 
affiliated with Education <22/l), the Social Sciences (20'/.), or the 
Physical Sciences (6/C). 

It would be a mistake to construe participation in this research 
project as implying that a person agrees with all the findings. Thus, 
where consensus is reported a minority of panelists hold divergent views. 
Where near unanimity is reported a same panelists may may not be in full 
accord wiLh hew the specifics are expressed. In the end, however, after 
reviewing the draft Delphi findings presented in Round 6, only one of the 
f<jrty-i5iK explicit-.ly opted to be listed as a participant only, but not as 
*nupportLng the document. 

These Delphi findings Hk an important moment in time. It is a 
moment when the efforts of forty-six experts possessing special 
experience and knowledge in matters relating to CT converged with a view 
toward discovering if same measure of general Accord could be found. As 
we move from t!ie ^successes of the eighties into the decade of the 
ni.ietie'ri, the person'5 who participated in this project hope that the 
finding'-i of ewper t consensus reported herein will advance critical 
Ihinking cttid hyip shape the future of CT instruction and CT assessment. 
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AfPiBMPIX A 

Commercially Available CT Asseaament Tools 



Prepared for the APA Committee on Pre-College Philosophy 
Delphi Research Project on CT Assessment 

The Education Testing Service <ETS) Academic Profile Test measures 
the academic abilities of CT, reading, writing, and using mathematical 
data, all Mithin the content of three major academic areas — 
humanities, social sciences and natural sciences. This is a multiple 
choice instrument with an optional critical essay (locally scared). It 
assumes that students have completed most or all of their general 
education — that is, completed the sophomore college year. 

ETS also has sections targeting logical reasoning and analytical 
thinking on several of its i^idely used instruments such as the (a) Law 
School Admissions Test, (LSAT) "Logical Reasoning" section, (b) Graduate 
Record Examination, (GRE) — General Test, the Analytical section, (c) 
Advancecj PU^cement Test, subject-matter based CT questions, (d) National 
Assessment of Educational Progress, higher order thinking and 
Idboratory-based questions, (e) Foreign Service Test, in-basket portion, 
^nd (f) Graduate Record Ewamination — Advanced Test in Philosophy 1972- 
19S2. ETB is adding a section of Critical Reasoning Questions to the 
Graduate Management Admissions Test <6MAT). ETS has constructed 
branching tests of information-seeking and decision-making, some paper — 
and-pencil and some on computers. An example is the clinical practice 
test prepared for the National Board of Respiratory Care, Shawnee 
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Missian, Kansas. 

Stephen Norris and R. King, through the Institute for Educational 
RKjsearch and Development at Memorial University o-f Newfoundland, has 
developed t^^e Test on Appraising Obsert/stions 1983. Using the backdrop 
of a common but fictional situation subjects are asked to judge the 
relative credibility of the claims made by various characters. 

The American College Testing Program (ACT) in 1988 produced the 
"Collegiate Assessment of Academic Proficiency" (CAAP). ACT describes 
GAAP is designed to measure selected academic skills including reading, 
writing, o^thematics, CT, and scientific reasoning. Tne CT Test 
measures the ability to "clarify, analyse, evaluate, anw extend 
argurtients." The test is composed of passages commonly encountered in a 
postsecondary curriculum fallowed by multiple choice test questions. 

Paul M. Ramirez discusses the "Valett Inventory of CT Abilities" 
rVICTA) in The Reading Teachern vol. 41, Dac. 1987, page 348. 

THE NINTH MENTAL MEASUREMENTS YEARBOOK. (NMMY), lists commercially 
available tests in print along with reviews and research data. Many of 
these are also described a d reviewed by Norris and Ennia in their 
useful Evaluating CT. Midwest Publications, Pacific fri-ove, CA, 1939. 

«769 "Cornell Critical Thii>,king Test, Levels X and Z." (Ennis, MiUman, 
Tcii.Jo), 1961-19S3, Midwest Publications, Pacific Srdve, CA. 
Reviewed in Educatiortal and Psychological ileasurements 1983 
Vol. 43., pp. 1187-1197, by Modjeski and Michael. 
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#390 "Cnnis-Weir Argumentation Test, Level X: An Essay Test of Rational 
Thinking Ability," (Robert Ennis and Eric Weir) 1982, Illinois 
Thinking Project, University of Illinois, IJrbana, IL. 
Reviewed by Herbert Rudman, Michigan State, in NMMY. 
#391 "Ennis-Weir Critical Thinking Essay Test: An Instrument -for 

Testing/Teaching." (Robert Ennis and Eric Weir) 1983, Midwest 
Publications, Pacific Srovu, CA. 
#1347 "Watson-Glaser Critical Thinking Appraisal" 1942-80. Described 
and reviewed by two persons in the NMMY. many citations of 
other research regarding this instrument. 
#751 "New Jersey Test of Reasoning Skills," 1983, Virginia Shipman, 
Institute for the Advancement of Philosophy for Children. 
#1250 "Test of Inquiry Skills" 1979, Australian Council for Educational 
Research. For junior high grades, this test purports to 
evaluate a range of research, study and critical thinking 
skills in the sciences. 
#1061 "Rass Te-3t of Higher Cognitive Processes" <Jahn Ross and Catherine 
Ross) 1976-79, Academic Therapy Publications. For grades 4-6, 
this test includes sub-scores on analogies, deductive 
risa-soning, missing prewdses, questioning strategies, and 
relevance of Information. 
#1248 "Test of Cognitive Skills" 1981, McGraw Hill. For grade levels 2- 
12, this test includes ^ub-scares on sequencing, analogies, 
nienory, and verbal reasoning. 
#122 "Basic Skills Asstfssment" 1977-81, McGraw Hill. Included in the 
reading p-Ackaye is a sub-score on inference and evaluation. 
In the writing package is a sub-score on logical evaluation. 
#1269 "Test of Problem Solving" 1984, LinguiSystem Inc. For ages 6-12, 
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this tests a child's thinking and reasoning abilities critical 
to events of everyday life. It includes sub-scorec. on 
explaining inferences, determining causes, negative why 
questions, etc. 

#272 "Corrective Reading Mastery Test" 1980, Science Research 

Associates, Inc. Designed to measure the effectiveness of 
corrective reading programs, this test includes sub-scores on 
deductions, classifications, analogies, inductions, statement 
inference, hypothesis/evidence. 

#1302 "Deductive Reasoning Test" (J. M. Verster) 1972-73, National 

Institute for Personnel Research, South Africa. Focuses on 
syllogistic problems and designed for for candidates for 
graduate scientists and higher professions. 

#1010 "PSI Basic Skills Test for Business and Industry" 1981-1982, 

Psychological Services Inc. Includes sub-scores on problem 
solving, decision making, reasoning and classifying. 

#106 "Ball Aptitude Battery" the Ball Foundation. Used to tests persons 
for occupational placements, this instrument includes sub- 
scores on inductive reasoning, analytical reasoning, idea 
fluency, and shape assembly. 



. APPENDIX B 

Critical Thinking Bibliogyanhy wi-hh E mphasis on Assesame nt. 

Prepared for the APA Committee on Pre-College Philosophy 
Delphi Research on Critical Thinking Assessment 



Adler, H. , "Why 'CT' Progracib Won't Work," Education Heek , Sept. 1986. 

Annis, David B. and Annis, Linda, "An Empirical Study of the Impact of 

Philosophy on Students' CT Ability," Teaching Philosophy, v3, pp. 145- 
152, 1980. 

Arons, Arnold B. "CT and the Baccalaureate Curriculum," Liberal Educatiorif 
v71, n2. Summer 1985. 

I "Achieving Wider Scientific Literacy," Daedalus, Journal of the 

Auerican Acadewy of Arts and Sciences, vll2, n2, p91-122, Spr. 1983. 

Azima, Kiavach, and Henry, Rebecca, "Teaching Students to Reason: An 

Application of Piagetian Psychology to College Teaching No. 7&," Learning 
and Evaluation Service, Michigan State University, East Lansing, MI, 40p, 
1980. 

Baker, P. J., "Learning Sociology and Assessing CT," Teaching Sociology, vS, 
P325-363, 1981. 

Barun, Joan B., and Sternberg, Robert J., Teaching Thinking Skills: Theory and 
Practice, W. H, Freeman Publishing, 1987. 

Bangort-Drowns, Robert L., et.. aJ., "Individualized Systems of Instruction in 
Secondary Schools," Revien of Educational Research, v53, n2, pl43-58, 
Summer 1983. 

Beck, Ronald A., A Guide to Cr iter i on-Referenced Test Construction , Johns 
Hopkins University Press, 1984. 

Beyer, Barry K. , "Improving Thinking Skills Defining the Problem," Phi 
Delta Kappan, v65, n7, p486-90, March 1984. 

1 "Improving Thinking Skills — Practical Approaches," Phi Delta 

Kappan, v6S, nB, Apr. 1984. 

I "Practical Strategies for the Direct Teaching of Thinking," in 

Developing Hinds: A Resource Book for Teaching Thinking, Arthur L. Costa, 
ed., Association for Supervision and Curriculum Development, Alexandria, 
VA, 1985. 

I "A Suggested Format for Testing Thinking Skills," Social Science 

Record^ v24, nl, p3-5, Spr. 1987. 

I Practical Strategies for the Teaching of Thinking, Boston, Allyn and 

Bacon, 1987. 
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Blatz, Charles V., "Contextual i sm and CT: Prograflimati c Investigations," 
Educational Theory, v39, n2, 19B9i 

Block, R. A., and Taylor, S. V., "Cognitive Skills.- Enhancement and Assessment 
Issues, Presented to the American Psychological Association, Toronto, 
Canada, 1984. 

Blumberg, Fran, et. ai, A Pilot Study of Higher-Order Thinking Skills 

AssessBent Techniques in Science and Hathewat ics — Part 1 and Pilot- 
Tested Tasks -- Part 11, Final Report, National Assessment of Educational 
Progress, Princeton, NJ, Nov. 1986. 

Bransford, J. D., et a/, "Teaching Thinking: Evaluating Evaluations and 

Broadening the Data Base," Educational Leadership, v44, p67-70, 1986. 

Branson, Stimmann Margaret, "CT Skills -- A Continuum for Grades 3-12 in 

History/Social Science," Social Studies Revien, v25, n2, p24-32. Winter 
1986. 

Brandt, Ron, "On Philosophy in the Curriculum: A Conversation with Matthew 
Lipnan," v46. Educational Leadership , p 38, Sept. 1988. 

Braungart-Blooro, Diane S. , "Assessing Higher Thinking Skills through Writing," 
lOp. Presented at the Annual Meeting of the American Educational Research 
Association, San Francisco, CA, April 1986. 

Brown, J. L., "On Teaching Thinking Skills in the Elementary and Middle 
Schools," Phi delta Kappan, v64, p709-714, 1983. 

Eryden, David P., "What Do Law Students Learn? A Pilot Study," Journal of 
Legal Education, v34, n3, p479-506, Sept. 1984. 

Carlson, E. R. , "Implications of Cognitive Theory and Research for Teaching 
CT, Presented to the Annual Meeting of the American Psychological 
Association, Toronto, Canada, 1984. 

Chaffee, John, "Viewing Reading and Writing as Thinking Processes," 9p, 
Presented to the Annual Meeting of the American Educational Research 
Association, Chicago, IL, Mar-Apr. 1986. 

Chance, Paul, Thinking in the Classroot: A Survey of Programs, NY, Teachers 
College Press, 1986. 

Cierzniak, Susanne Lipetska, The Question of CT: An Annotated Bibliography , 

64p. Exit Project, University of Indiana at South Bend, ED 260 069, April 
1985. The specific focus of this work is CT in the secondary schools. 



Cornbleth, Catherine, "Assessing Skills and Thinking in Social Studies," 
position paper prepared for Study Group on the national Assessment 
Student Achievement, and cited in Appendix B of their report, "The 
Nation's Report Card" (TM 870 049), Journal announcement: RIEJULB? 

Costii, Arthur L., ed. Developing Hinds: A Resource Book for Teaching 

Thinking,, Association for Supervision and Curriculum Development, 
Alexandria, VA, 1985. 
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1 "Thinking: How Do Me Know Students are Setting Better at It?" ms.,' 

Dept. 0^ Education, Sacraniento State University, CA. , 1989, ' 

Costa, Arthur L. , and Marzano, Robert J., "Teaching the Language Thinking." 
Educational Leadership^ v45, p29, Oct. 1987. 

O'Angelo, Edward, The Teaching of CT , Amsterdani, B. R. Gruner N. V., 1971. 

Deshniukh, M. N. , "Teaching the Unteachable: Some Pedagogical Considerations 
of Creativity," Psycho-Lingudf vl5, nl, p33-40, 1985. 

Dewey, John, Hoti He Think, D. C. Heath and Co., Boston, 1933. 

Drake, James A., Teaching CT, Danville, IL, Interstate Printers and 
Publishers, 1976. 

Duck, Lloyd E. , "Seven Cardinal Principles for Teaching Higher-Order 
Thinking," Social Science Record, v24, nl, p3-5, Spr. 1987. 

Elman, Sandra E. , and Lynton, Ernest A., "AssessniBnt in Professional 

Education," 24p. Presented to the National Conference on Assessment in 
Higher Education, AAHE, Columbia, SC, Oct. 1985. 

Ennis, Robert H. , 'ft Concept of CT," Harvard Educational Revie*i, v32, nl, 
1962. ' 

— » "Operational Definitions," Aterican Educational Research Journal, vl. 
n3, plB3-201, May 1964. ' ' 

"Testing for CT: State of the Art," American Educational Research 

Assoc., San Francisco, CA, 196B. 

» "Rational Thinking and Educational Practice," in Philosophy and 

Education, Jonas Soltis (Ed.), Eighteenth Yearbook of the National 
Society for the Study of Education, Chicago, NSEE, pl43-183, 1981. 

1 "Goals for a CT/Reasoning Curriculum, Illinois CT Project, U. of 

Illinois, Champaign, IL, 19B4. 

- ' "Problems in Testing Informal Logic CT Reasoning Ability," lnfor»al 

Logic v6, nl, p3-9, 1984. 

' Logical Basis for Measuring CT Skills," Educational Leadership, 

v43, p44-48, i985. 

1 "A Logical Approacn to Measuring CT Skills in the Fourth Grade," 

Illinois CT Project, Champaign, IL, draft, April 1985. 

' TaKonomy or CT Dispositions and Abilities," in Teaching for 

Thinking, Joan Baron and Robert Sternberg (Eds.), Freeman, New York. NY. 
1987. ' ' 

1 "A Conception of CT with Some Curriculum Suggestions," Hessletter on 

Teaching Philosophy, American Philosophical Assoc., pl-5, Summer 1987. 
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"A Bibliography oi Testing CT," CT We«?, Center ior the Reasoning 



Ennis, Robert H, Gardiner, William L, et ai, Cornell Class Reasoning Test, 
University of Illinois, 1964, 

, Cornell Conditioml Reasoning Test Illinois CT Project, Champaign, 

IL, (date ?) 

Ennis, Robert H., and Millman, Jason, Cornell CT Test Level Midwest 
Publications, Pacific Grove, CA, 1985. 

, Cornell CT Test Level Z, Midwest Pub., Pacific Grove, CA, 1985. 

Ennis, Robert H. , and Weir, Eric, Ennis-Ueir CT Essay Test^ Midwest 
Publications, Pacific Grove, CA, 1985. 

Ennis, Robert H. , and Norris, Stephen P., "CT Testing and Other CT Evaluation: 
Status, Issues, and Needs," in Issues in Evaluat ion ^ Algina, James, 
(Ed.), Ablex Press, New York, NY, 19BB(?). 

Facione, Peter A., "Toward a Theory of CT," Liberal Education^ v70, p253-261. 
Fall 1984. 

_, "Testing College Level CT," Liberal Education^ v72, p221-232. Fall 
l986. 

, "Teaching about Fallacies," Teaching Phi losophy , viO, pp. 211-217, 

Summer 1987. 

, "Some Current Concerns Regarding HOTs Assessment," CT Wews, v7, n2/3, 

p3 & 9, 1988. 

, "Assessing Inference Skills," ERIC Clear Inghouse on Tests, 

Heasureaents and Evaluat ion ^ TM 012917, Mar. 1989. 

, "Reasons and Premises," CT Hens v7, n4, pi, 7f, 1989. 

, "A Quick List of CT Tests," CT Wews, v7, n4, pl2, 1989. 

"Some Definitions of CT," CT We«5, v7, n4, plO, 1929. 

, "Strategies far Multiple Choice CT Assessment," in CT at Colleges and 

Un iversit ies ^ David Hitchcock, (Ed.), Vale Press, Newport News, VA. , 
f orthcoffli ng. 

Follman, John, "Contemporary CT Bibliography," CT We«5, Center for the 
Reasoning Arts, CSU Sacramento, v6, n2, Nov, -Dec. 1987. 

Fraser, Berry, Test of Inquiry Skills^ The Australian Council for Educational 
Measurement, Hawthorn, Victoria, 1979. 

Frederiksen, Norman and Nard, William B. , "Measures for the Study of 
Creativity in Scientific Problem Solving," Applied Psychologi cal 
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Measureweui, v2, pl-24, 1978. 

Fritz, Paul A., and Weaver, Richard L. II, "Teaching CT Skills in the Public 
Speaking Course: A Liberal Arts Perspective," 36p, Presented at the 
Annual Meeting of the Speech Coflimuni cati on Assoc., Chicago, IL, Nov. 
1984. 

Sardner, H. , Traie? of Hind: The Theory of Multiple Intelligences, Basic 
Books, New York, 1983. 
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APPENDIX G 
The Delphi Research Let ters 



Over course of this research sixteen "Delphi Letters" were 
sent to the experts participating in the APA Committee on Pre- 
College Philosophy Delphi Research Project on CT assessment. 
Eight letters constituted the specific interactive Delphi 
research rounds. The eight others were for purposes of planning, 
clarifying procedures, providing information, and sustaining 
involvement. 



Feb. 


11, 


1988 


ROUND 1 


Mar. 


1, 


1988 


Information and Plans 


Mar. 


14, 


1988 


ROUND 2 


Apr. 


14, 


1988 


Plans and Procedures 


May 


4, 


1988 


ROUND 3 


May 


18. 


1988 


Procedures and Involvement 


June 


28, 


1988 


Information and Involvement 


Sept . 


1. 


1988 


Procedures and Information 


Sept. 


23, 


1988 


ROUND 4 


Nov. 


22, 


1988 


Plans and Involvement 


Feb. 


7, 


1989 


Information and Involvement 


Feb. 


28, 


1989 


ROUND 5A 


Mar. 


6, 


1989 


ROUND 5B 


Mar. 


10, 


1989 


ROUND 5C 


May 


9, 


1989 


Information and Involvement 


Sept. 


25, 


1989 


ROUND 6 



California State University, Fullerton 

Fullerton. California 92634 



Department of Philosophy Feb. 11, 198 8 

(714) 773-3611 



Dear Colleague, 

Gary Matthews, Chair of the APA Committee on Pre-College 
Philosophy, asked me to head up an ad hoc committee on testing 
critical thinking. To get the project" rolling, he suggested severa 
names of people interested in the question of ^-^w to validly and 
reliably test critical thinking skills. I adae^i a few more. Here are 
some of the particulars. 

1) By using a modified Delphi approach, I think the necessity for 
actual meetings can be largely, if not entirely eliminated. Committee 
members' contributions will involve sending their reasoned and timely 
responses to questions, given their particular background and 
expertise. 

2) There are, no doubt, a great number of other people who are 
interested in the question of testing critical thinking and who have 
valuable expertise which would help us in in dealing with this 
question. They should also be invited to participate. 

3) Since our charge is rather vague, 1 propose that among the 
first things we should do is agree on priorities. To do this, using 
the Delphi process, let me lead off with some assumptions and 
questions: 

First assumption; Most of the members of our group will come ac 
the issue of testing critical thinking with the orientation of 
philosophers or logicians who teach at the posU-secondary level, 
rather than as K-12 educators, psychologists, or personnel 
directors (all of whom also have legitimate theoretical and 
practical interests in assessing critical thinking) . Given the 
interests of the Amer-'can Philosophical Association, this 
acceptable . 

Second assumption: Critical thinking can be defined operationally 
to the extent that it can become a dependent variable in a valid 
and reliable assessment tool. 



QUESTIONS: 

1. Do you agree with the two assumptions? How would you 
amend/clarify them? Why? 

2. To which educational level (from Kindergarten through post- 
Baccalaureate) should the committee give priority? Why so? 

3. After looking the attached preliminary list, whom else 
would you recommend be added to our com.mittee? 

The CaMorma Slate University 
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For the Delphi process, which is very interactive, to function 
optimally, reasonably quick turn around is needed. Let's target a 
Feb. 29 postir.ark. Is that possible for you? 



Please send responses, suggestions, comments, etc. to: 

Ad Hoc APA Committee on Critical Thinking Testing 

c/o Dr. Peter A. Facione 

Professor of Philosophy and Education 

Dept. of Philosophy 

California State University, Fullerton 
Fullerton, CA 92634 



Like math and composition, at many colleges throughout the 
country CT is being built into the curriculum. For example, the 
California State University system, which enrolls hundreds of 
thousands of undergraduates, has implemented a system-wide critical 
thiiiking requirement as part of its general education package. If 
those of us who teach critical thinking were able to agree on a way or 
ways it could be tested, what a positive contribution that could be to 
the quality of that curriculum. 

I sincerely hope you will agree to become an active participant 
in -what promises to be a most interesting and important effort. 

I've included some reference material, from The Ninth M enta l 
Measurements Yearboo k regarding published instruments which purport to 
neasure critical thinking and/or related cognitive skills. 



You;«^ since rely , 




Pete Facione 



C C • 



Gary Matthews, University of Massachusetts, Amherst 
Steve Tigner, University of Toledo, Ohio 



ATTCHMSNTS: Preli.-ninary List of Committee Members 

Quick Survey of Published Instrurr.ents 



TO BE DEVELOPED: Bibliography on Testing Critical Thinking 
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California State University, Fullerton 

Fullerton. California 92634 



Dopa.lment of Philosophy 
(714) 773-3611 

March 1, 1988 

Dear Colleague, 

Early responses to the first Delphi round are very encouraging! 

Some of you know what I'm talking about, others, new to the effort, 
need to be brought up to speed. Let me back track a bit and explain. 

I am writing to you because you were nominated by one or more of 
your colleagues as a person interested in the question of testing 
critical thinking. You are being invited to participate in the work of 
an ad hoc American Philosophical Association subcommittee concerned with 
the problem of testing critical thinking. In its boldest form, our aim 
is to find ways to validly and reliably test critical thinking, or find 
solid reasons why such a goal is not achievable. Using the Delphi 
process, I will serve as coordinator of the effort. At the moment we 
are very near the beginning of Phase 1. 

Please review the preliminary plan outlined below and decide to 
becorae actively involved in what promises to be an intriguing effort to 
shed light on an important pedagogical and profession concern. 



ad hoc APA Sub-Committee on Testing Critical Thinking 

Draft Preliminary Plan 



Phase 1: Start the Inquiry 

The five objectives of this phase are; 

(a) Initiating the Delphi process. This is a method of achieving 
reasoned consensus among a group of experts with regards to a given 
problem or issue. The core of the strategy is to maka inquiries, 
g-ther each expert's responses and their reasons, then summarize 
and share those with the group. After "hearing" what other experts 
think, people have the opportunity to refine their responses or 
defend those responses. The interactive rounds continue until 
reisoned consensus is achieved (or communications break down). 

(b) Developing the "List of Experts" who will take part in this 
inquiry. Many of you were nominated by those who replied to my first 
letter, (2-11-88). That letter was sent to an original group of 
about twenty-five experts and interested persons suggested by 

the APA comnittee that conceived of this project and asked ine to 
coordinate it. A: any time if someone is no longer interested in 
continuing, just drop me a note. I plan to send out updated 
rosters of participants periodically. Ve are now up to fifty. 
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(c) Developing a bibliography on testing critical thinking. Work 
on this is progressing well. Many of you have sent me items to 
include, and I appreciate that. I will send you a copy of the 
bibliography later this semester. 

(d) Planning the subsequent phases in our process of responding to 
the general question of whether or not critical thinking (whatever 
CT is) can be validly and reliably tested at some educational 
level or levels. If the consensus is "Yes, at level X/' then we 
will focus on the question: How? if the consensus is "No, at least 
not at level X," then wo will focus on, "Why not?" 

(e) Agreeing on basic assumptions. 

In the 2-11-88 letter two assumptions were put to the group: 

Assumption §1: "Most of the members of our group [of experts 
participating in the Delphi" process] will come at the issue of 
testing critical thinking with the orientation of philosophers 
or logicians who teach at the post-secondary level, rather 
than as K-12 educators, psychologists, or personnel directors 
(all of whom also have legitimate theoretical and practical 
interests in assessing critical thinking) . Given the 
interests of the American Philosophical Association, this [is 
an] acceptable [orientation]." 

Assumption §2: "Critical thinking can be defined 
operationally to the extent that it can become a dependent 
variable in a valid and reliable assessment tool." 

People were asked (1) if they agreed with the two assumptions 
as stated, or whether they would reject thera or rephrase thera 
somehow. Naturally, people were invited to explain why. They 
vere also asked (2) to identify the educational level (K-post 
baccalaureate) to which our committee should give priority, 
and why they would recommend that level. 

Both questions have generated controversy, as you will see in my 
next letter. If you haven't h.id the opportunity to respond to these 
questions, you will be invited to respond when the first round of the 
Delphi is reported back to you. Delphi is not about vote-counting, it 
aims at reaching agreement on the basis of reasons and common 
assumptions. In the Delphi method people are supposed to share their 
premises, not just their conclusions. 



As conceived at the moment, our work can be divided into 4 phases. 
These are not in stonel I welcome your suggestions, amendments, 
alternatives, etc. VE WILL USE THE DELPHI PROCESS TO AGREE ON OUR PLAN 
OF INQUIRY. Because the plan should be. amended as a result of your input, 
only goals, not detailed objectives, have been developed so far for 
the next three proposed phases. 
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Phase 2: Define "Critical Thinking" 

The goal of this phase is to define "critical thinking" with 
sufficient clarity and precision to ask and answer the 
question of whether or not CT can be tested. Is CT 
f undaipentally a set of skills, concepts, procedures, 
attributes, behaviors, outcomes, dispositions, aptitudes, or 
what? Even if we cannow reduce CT to an equivalent 
operational definition, how might we express what CT is with 
sufficient operational precisions to permit us to justifiably 
■ infer things about the relative CT abilities of students? 

Phase 3: Recommendations 

The goal of this phase is to communicate our findings about 
what CT is and whether there is an adequate way of 
characterizing CT operationally so as to permit its being 
tested as some educational level. Depending on our results in 
Phase 2, we will recommend either that programs aimed at 
testing CT be abandoned, orjthat they be focused in certain 
ways. If this is the direction Phase 3 takes, then we will 
also try to come to consensus on recommendations regarding the 
relative importance of different kinds of CT sub-skills and 
possible strategies for accessing and measuring those sub- 
skills. 

Phase 4: Design and Validation of Model Testing Strategies 

Contingent on the results of earlier phases, the goal, if it 
were considered achievable in principle, would be to construct 
and evaluate different approaches to testing CT at some 
appropriate educational level or levels. We might find 
ourselves breaking into sub-committees to achieve this goal, 
although all work will have to be guided by the agreements 
reached in earlier phases and as well as by the special 
expertise of those who understand the intricacies of 
designing, piloting, norming and validating educational tests 
at specific educational levels. 



As I mentioned, you have been nominated as person who might be 
interested in this project and could make a strong- contribution to the 
work of this ad hoc sub-committee. I hope you will agree to participate 
actively, because, as you must know, tne quality and utility of our 
effort is directly related to the involvement of concerned persons like 
yourself. 



Sincerely, 




Pete Facione 
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California State University, Fullerton 

. Fullorton. California 92634 



Ooparlmantof Philosophy 
(714) 773-3611 

March 14, 1988 

Dear Testing CT Colleagues, 

Let's give Phase I, R ound 2 a shot! 

Than)cs for your responses to the first round of questions. Nineteen of the 
twenty-six or so who received the original 2-11-88 letter for Round 1 were able 
to respond. Round 2 invites everyone (which now includes just over 50 people) 
to review the results of Round 1 and coaaent on the agreements and controversies 
that are emerging. 

Round 1 focused on three issues: (1) The coaposition of our ad hoc 
committee in view of the interests of the Xierican Philosophical Association, 
(2) the assumption that critical thinlclng can be operationally defined, and (3) 
the educational level to which we should give priority. 




In regard to the composition of our coamittee, we are in decent shape, 
particularly since our group has been greatly expanded as per your 
recommendations. In regard to an operational definition of CT, we generally 
agree on the possibility but many would add various caveats. In regard to the 
educational level to which we should give priority, we have disagreements. 



The following pages cover each of the three questions in turn. You'll find 
restatements of the original questions and several representative quotes and 
summaries of your comments. AFTER EACH SUMMARY, A SECOND ROUND QUESTION WILL BE 
PUT TO YOU. The new questions take the form of stating a position and asXing 
your opinion, now that you have had an opportunity to consider what our 
colleagues have to say. In all, there are three new questions. IF YOU COUL D 
GET YOUR RESPONSES TO ME VITHIN TEN DAYS OF RECEIVING THIS LETTE R, THAT WOULD BE 
^^^^Ti, (I wish we all had electric mail, or unliT.\ted phone budgets, but...) 



Several people noted that our task was huge, yet were willing to give it a 
try. In contrast, one person wrote a major critique of the entire enterprise. 
This person argued that trying to test CT was a serious mistake. So that his 
opinion is not lost in the shuffle, at the end of this package I have provided 
extensive quotations from his letter. If you find yourself in agreement with 
his views, then let me know and we will take up any "prior questions" we .Tiust. 
.If you don't agree, then we will press on along the path we are charting for 
ourselves. 



Think you in advance for your partUipation. 

..60 . ' , 
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PHASE I, ROUND 2, FIRST ISSUE 
CURRJ:NT STATUS: CLOSE TO CONSENSUS 

COMPOSITION OF THE AD HOC COMMITTEE: Ve yere asked if we agreed with the 
assumption that, although most of the aeabe.'s of our ad hoc committee would come 
at the issue of testing with the orientation of philosophers or logicians who 
teach at the college level, this orientation still would be acceptable in terms 
of the interests of the American Philosophical Association, 

Almost everyone agreed, however some qualified their responses in terms of 
our collective professional interests and abilities, or in terms of educational 
level to which we should give priority. Here are representative responses: 

"I agree." "I have no problems with this assumption." 

"I don't see why this is a problem. First we're concerned about students 
acquiring the thinking skills required for college work,... Second, we're 
concerned that they learn the standards of good reasoning; I do not believe we 
need to know a lot about psychology to .-achieve this purpose." 

"I agree, with reservations. We need to avoid tunnel vision. It is 
acceptable that most members be philosophers, but there should be a generous 
sprinkling of 'outsiders' for the insights they will bring and to give our 
findings greater credibility outside the APA." 

"We are what we are! This is an appropriate place to begin. We are 
starting from what we know best and with what we can deal with most easily. 
This is not to suggest that we shall forever ignore other orientations, or that 
we really know that we can define all aspects of CT operationally." 

"I agree, this is acceptable; but it is unnecessarily narrow, since so 
many of the tests are created by cognitive and educational psychologists, I 
think some of them should be included..." 

"It should not be too quickly assumed that those who teach at the post- 
secondary level are therefore knowledgeable and competent with regard to testing 
at the elementary school level." 

"I agree, but we should make a r-erious effort to inform ourselves of 
approaches to CT in pre-college and non-academic settings... Assuming our 
primary focus is everyday reasoning skills, we should not allow college CT 
instruction to be fundamentally different from pre-college CT instruction nor to 
become idiosyncratically colored by our own traditions. 

One person disagreed but did not give a reason. And one urged "Pete, get a 
proof-reader!" 

RPIip 2^ QyE_Sj.IO» OflE: In view of the above comments, and in view of the 
additional names added as the result of your recommendations, can we agree that 
the ad hoc co.r.nittce. as listed on the attachT.ent, is sufficiently well- 
constituted for us to move on with our main task? As you can see, it still has 
its original orientation toward philosophers teaching at the college level, but 
it also includes several people from other relevant disciplines backgrounds, 
including psychology and education. 
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PHASE I. ROUND 2, ISSUE 2 
CURRENT STATUS: AGREEMENT SEEMS ACHIEVABLE 

OPERATIONALIZING CRITICAL THINKING: Ve were asked if we agreed with the 
assuwption that CT can be defined operationally to the extent that it can become 
a dependent variable in a valid and reliable assessment tool. 

Here, too, most people agreed and were ready to get on with the work. Yet, 
some crucial ambiguities, concerns and caveats emerged. Here are some 
responses. 

"I expect we will argue about the details of any definition proposed, but I 
do not object to the assumption that we shall need some such instrument if we 
are to get any comparisons of interest." 

"This is a tautology because of the to the extent phrase. Perhaps this is 
how we should leave it..." 

"Sounds ok. ...I'm not a statistician, so I'm not quite sure what 
dependent variable means — but if you 'are asking whether CT can be tested, 
then, yes, I agree!" 

"I don't understand what this assumption is supposed to mean!" 

• "I would agree only if we amended it to say at least some components of CT 
can be defined operationally to the extent that they can become dependent 
variables in a valid and reliable assessment tool... I do not accept as analytic 
the proposition that CT can be defined operationally... I think some important 
aspects of CT, such as making judgment calls and weighting nuances may resist 
operational definition." 

"As the term operational definition is generally used by philosophers and 
education researchers, I do not think CT can be operationally defined... but I 
do think that part of the operational spirit can be employed in formulating 
reduction sentences (that do not reduce!)." 

"I agree, but there will probably have to be a variety of sub-definitions 
because CT is not one thing, but many. It somewhat resembles IQ in that." 

"There are several definitions of CT floating around... Some lend 
themselves more to operational definition than others.... If we are to get 
anywhere, we will have to become clear in our own minds as to how CT is to be 
distinguished from other kinds of thinking..." 

"Is this a normative, definitional, conceptual, or planning assumption?" 

ROUND 2_L. QUESTION TWOi Without hanging ourselves up on the word "operations," 
can we agree that: (1) Even if CT cannot be reduced entirely to an equivalent 
set of operations [or performances, behaviors, processes, outcomes, or skills,] 

(2) it is possible to conceptually analyze CT so as to describe a set of 
relevant and important CT operations, such that (3) using these descriptions, 

[competent] investigators could, on a consistent basis, gather sufficient 
evidence to draw conclusions, with high degrees of confidence, regarding the 
relative CT abilities of a group of people, (everything else being equal, of 
course] . 
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PHASE I, ROUND 2, QUESTION 3 
STATUS: CONTROVERSY 

PRIORITIES: Ve were asked which educational level (kindergarten through 
post-baccalaureate) should be the priority for our comaittee. And why so? 

Responses were split. Here are some representative examples: 

A person with considerable experience in the area of CI testing wrote: "I 
think we should concentrate on high school at first, since this is a 
subcommittee of the pre-college committee, all [levels] above that [are] ruled 
out. Furthermore, the younger the population, the more difficult the problems. 
Let's start with the easiest ones first — and they are very difficult." 

By contrast, it was argued, "Priority should be given to the post-secondary 
level. One should examine end-products first, and then work backwards if 
needed. Find out first if the car doesn't run before attempting to determine 
where the problem is. If a good test of CT revealed no CT deficiencies on the 
part of graduating seniors — (no doubt a counter-factual assumption) — then I 
would think the APA might not wish to pursue the issue down in K-12." 

Noting that we are a sub-coaaittee of the APA pre-college coamittee one 
person argued: "We must give priority to K-12; that is our mission." 

However, the person who will assume the chair of the APA pre-college 
committee for the next three years wrote: "It makes sense to start by playing 
from the APA membership's greatest experience and strength, which is surely 
college freshman level logic." 

Taking note of the interests of the APA, one person argued, "Since our ad 
hoc committee is convened within the structure of the APA, our focus should 
perhaps be primarily on the improvement of post-secondary education..." 
However, this person also suggested, "... that our assessment tool should be 
usable in secondary schools as well as at the post-secondary level..." 

Some people did not offer an opinion, but did note important distinctions. 
For example: "There are really two areas. One is the whole K-12 integration of 
thinking skills into the curriculum. The other is the single CT course, 
typically the approach followed in post-secondary education. The single college 
CT course offers exceptional opportunities for measuring gains in thinking 
skills, while the effort to incorporate thinking skills into the [K-12] 
curriculum may offer much greater potential for actually improving student 
skills." 

Another person, experienced in the pre-college arena, wrote: "It may be 
necessary to think of four tests, one for grades K-3 (one should not have high 
expectations for reliability at this level); one for grades 4-8 (the level at 
which testing might have the maximum impact, even though the maximum impact for 
the teaching ot CT might be at K-3); one for grades 9-12, and one for 13-16." 

Some were tentative: "Perhaps we should give priority to CT at the collage 
level, at least to start with, since the large majority of APA members teach at 
the college level. Later we might wish to broaden our focus." 
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SotQe were direct, "College and university level." 

Others were focused, but concerned not to overlook anything important, 
"College level — but someone ought to look at the high school level." 

One person declared for the college freshman level and ?rgued against going 
any higher saying, "There are few if any thinking skills posses'-ed by people 
beyond this level not also possessed by well-prepared college freshman. 
Graduate school and professional life chiefly consists in the ability to 
persistently apply these skills in more and more recondite subject matters. 

Some narrowed the range, but still left us with a choice: "I would say 
grades 9-12 and freshman/sophomore level in college." 

Another argued we should give priority to the introductory baccalaureate 
level saying, "First, it is the area where most philosophy departments have 
numerous classes actually being taught. Second, it is taught at a level which 
will have the raost connections in other areas and at other levels." 

ROUND 2t_ Q UESTION THREE ; To get started let's give priority to the college 
freshman/sopboaore level. Do you agree? If not, is your disagreement based on 
pedagogical and theoretical concerns or on concerns relating to our charge as a 
sub-committee of the APA committee on pre-college philosophy? 



Please try to qot, your responses to me within ten days. Thanks. 



G4 
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PHASE I, A CHALLENGE RAISED AGAINST THE ENTERPRISE 



One person wrote: "I would like to comment that teaching CT skills... is a 
far more significant matter than testing for them. Since CT is not an inherited 
trait, as is intelligence, the quality and extent of the CT is dependent on and 
proportional to the degree to which chiMren have learned to, or have been 
taught to, think critically about their experiences and knowledge. 

"Our educational system ... is an abysmal failure. Most students are 
unable to recognize assumptions, not alone question or examine them. ... With 
rare exceptions, they show an almost total absence of recognition of even the 
simplest of logical/mathematical/linguistic/philosophical/scientific facts and 
concepts needed to be able to think critically. They have been so nurtured in a 
world of superficial "education" ... that to think of testing them on the basis 
of that "education" is an exercise in futility or at best an attempt to 
determine how inadequately they think critically as opposed to how much — which 
if taken literally amounts to the same thing. 

"I am willing to contribute my expertise to the teaching of CT. I have 
been doing so for over thirf years. But until I see considerably more evidence 
of students being able to think critically without such teaching, I see little 
point in testing them for the insignificant amount of critical acuity they may 
hav'e acquired haphazardly, 

"... Teaching CT must precede and supersede testing for CT. Testing for CT 
cannot be considered to be an enterprise separate from teaching it. Testing for 
such skills and concepts presumes prior teaching of them. Psychologists and K- 
12 teachers as well as other educators show interest in CT. The problem 
remains, however, that most of them have only superficial, naive, and 
conflicting concepts, of what critical thinking entails. Even we philosophers 
can't agree on what it is... 

"From my understanding of the term, testing for CT means testing based on 
what I teach CT is. ... Any tests that I would, and have designed, are 
predicated upon the version of CT I have taught. 

"My comments will undoubtedly reflect those which you will receive from ray 
colleagues equally concerned with the problems facing our educational system, 
particularly as they relate to the teaching of CT skills and concepts." 



REMINDER: If you believe there are issues (suggested by the above or otherwise) 
which our ad hoc sub-committee must address before we can move ahead, please let 
me know, on the other hand, the above challenge may represent a view which is 
not widely shared, or may raise questions which, in your view, do not fall 
within the scope of our work or do not warrant our attention at this time. If 
that is the case, then, for the present, no response on this item is necessary. 



Thanks again for your participation. 
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California State University, Fullerton 



Fullerton. California 92634 



Department of Philosophy 
(714) 773-3611 



April 14, 1988 



Dear Colleagues; 



Thanks for your help with the addresses. I also appreciate the 
notes, suggestions and other helpful comraents many of you are 
sending. 

So far 17 responses to Rjund Tvo have been received. It would 
help if we had more, particularly since it there may still be a 
split over Question Number 3, the one-'about which level to pursue 
first. Drop me a note, with your reasons and opinions. Thanks. 

One of the responses to Round Two proposed an alternative to 
actually trying to come up with our own CT assessment instrument 
— a goal some of us hope to achieve, but others of us are 
extremely skeptical about. This person suggested that we 
articulate the best list of CT skills we can, then let people go 
their own way with regard to building testing instruments. I'll 
expand on that idea when I surnnarize Round Two responses. I 
mention it here because perhaps we all should be chinking ahead 
and trying to chart the most reasonable path for ourselves. 

Some of you have electronic mail. I don't, at least not yet. 
So, I'll be using snail mail and telephone to try to reach you 
for particular questions or clarifications. If you want to phone 
in your views on the questions in Round T^o you can reach me at 
7H-773-3742 (office) 08:30-10:00 IIW or 09: 30-11: '0 TTh (PDT) . 
If those tiir.es are inconvenient, call the department secretary at 
714-773-3611 and leave your phone number. I'll get back to you. 

I will be attending the First National Conference on Assessing 
Thinking in Baltimore on May 6 and 7. This conference is 
sponsored by the Maryland State Department of Education and the 
Association for Supervision and Curriculuir. Development. Although 
plans are to discuss all educational levels, the participating 
organisations, (over 35 professional associations, centers and 
government agencies) are concerned primarily with K-12 education 
in some way or anothei". I'll report on what pron^is^s to be a 
most interesting gathering. 

Vithin the week you should receive two ic=r.s interest. One is a 
partial bibliography CT with emphasis on tescina CT, the other is 
an updated listing of sorr.e of the existing t-ists wluch purport to 
measure CT or closely related reasoning s>;ills. 
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The California Slaie University 



t 



6R 



Department of Philosophy 
(714) 773-3611 



May 4, 1988 

Dear Colleagues, 

^^^^°Pe this letter finds you happy and well. It's time for 
Round 3 of our Delphi — the round where we finally get to the 
heart of the matter: What is CT? Also, for those who are new 
to our effort, this letter includes a brie/: overview of who we 
are and what we are about. The last three pages suiiunarize our 
Round 2 results. Please send Round 3 responses by June 25. 

As a result of nominations in Round 1 and Round 2, sixty 
persons, including some of the most eminent names in the field 
are now invited to participate in this effort. 

By way of background, in January the American Philosophical 
Association Committee on Pre-College Philosophy asked me to chair 
an ad hoc sub-committee on testing critical thinking. Beginning 
with an initial group of APA nominees and asking them for 
additional recommendations, the "sub-committee" has grown to 
include people from a variety of academic disciplines and 
professional affiliations. Our unifying concern is in testing 
CT. However, we do not necessarily share the same 
conceptualization of what CT is nor do we necessarily agree on 
how It might best be tested. 

It is to resolve precisely these two things that we have 
undertaken the Delphi process. In Round 1 (Feb. 11, 1988) and 
Round 2 (Mar. 14, 1988) the focus was on establishing group 
membership and agreeing on preliminary working assumptions 
such as the assumption that CT could be opera tionalized to the 
extent that valid and reliable assessments of important and 
relevant CT skills could be made. Starting with Round 3 we will 
focus on what those CT skills are and eventually we will decide 
on recommendations regarding testing, based on any Delphi 
consensus we achieve. To assist with the conceptual work that 
must be done, I developed and circulated two items, a list of 
existing CT tests and a CT-Testing Bibliography (Apr. 19, 1988). 

_ If you want copies of any of materials mentioned or if you 
riine° ^ °^ mailing/membership list, just drop me 

Last week I presented a workshop on testing CT at Sacramento 
ijcate. At that time Perry Weddle agreed to publish the CT- 
testing bibliography and the list of CT tests in a fall issue of 
CT News. So, please get any corrections, additions, or deletions 
to me as soon as you con. AND HOW OH TO ROUIJD TJIREE ! 
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**** ROUND THREE **** 



QUESTION: WHAT OPERATIONS ARE CENTRAL TO CT? 
Response requested by June 25 . 

The sole task for Round 3 is for each of us to come up with 
that list of operations [or performances, behaviors, processes, 
outcomes or skills] which we understand to be at the core of the 
concept of Critical Thinking. 

Although many of us have published on this question, since 
there are potentially sixty in our group and since I have no 
assistants, it will be most helpful to me if you would take the 
time to distill your views and send a list of what you interpret 
the central CT operations to be, fell free to indicate which are 
the more general and which are the sub-operations . Naturally you 
are welcome to include justifications for the items on your list. 

In thinking about this, please keep in mind that in the two 
preliminary Delphi rounds we have narrowed our focus for now to 
CT "operations" understood as performances , behaviors . processes . 
outcomes or ski lls which could be tested validly and reliably at 
the college freshman/ sophomore level. But keep in mind that we 
very likely will extend the question downward to K-12 later. 

In Round 4, which 1 will initiate in Sept., you will be 
given combined lists and invited lo comment on the wisdom of 
excluding, retaining, or amending the descriptions of specific 
items. If the results of Round 3 are clean enough, Round 4 will 
also invite you to begin rating items in terms of how more or 
less important, crucial, central, integral etc. they are to the 
concept of CT. 

If you do not intend to respond to Round 3, for whatever 
reasons, please drop me a line so that I can keep track of 
participation levels. 

Matt Lipman suggested that we might have an excellent chance 
of working with the APA and the Assn. for Informal Logic and CT 
to secure the use of the Wingspread Conference Center. Please 
let me know if you think it would be productive to get together 
in that setting. How might a conference be organized to most 
effectively use our valuable time? What kinds of issues, 
problems, tasks might we address? What kinds of solutions or 
desirable results might we achieve by meeting which couldn't be 
achieved (at all or as well) using the Delphi? 

Since Round 3 asks the "big" question, please take the time 
to respond. 



Sincerely, 
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SUMMARY OF RESPONSES TO ROUND TWO 
Overall response rate: 23 of a possible 51. 

Ouestion 1 pj. Round 2 asked if cur sub-committee was sufficiently well 
constituted to move on with our task. As you can infer, the consensus answer 
was "Yes." A small number of additional names from psychology and education 
were recommended and strong cases for adding thep were made. That is how we 
moved up to sixty members — presuming the new nominees agree to join in. I'll 
send you an updated list soon. 

Question 2 of Round 2 asked if we agreed with this claim; 

"(1) Even if CT cannot be reduced entirely to an equivalent set of 
operations [or performances, behaviors, processes, outcomes, or 
skills,] (2) it is possible to conceptually analyze CT so as to 
describe a set of relevant and important CT operations, such that (3) 
using these descriptions, [competent] investigators could, on a 
consistent basis, gather sufficient evidence to draw conclusions, 
with high degrees of confidence, regarding the relative CT abilities 
of a group of people, [everything else being equal, of course]." 

There was sufficient consensus on this to move ahe.'^d. The majority of 
responses (18 were strongly to moderately positive, 2 were negative and 3 did 
not respond to this question.) 

However, to avoid misunderstanding, let us keep i.. mind that the above 
statement should not be interpreted to imply that construct validity can be 
determined strictly in an a priori manner (by simply coming up with our list) . 
Nor should the words "relevant and important" be interpreted to imply that we 
can come up with an exhaustive list. Nor should the purpose in (3) be 
interpreted to mean that we have set our sights on actually writing a CT test, 
or, for that matter come to any agreement about what recommendations regarding 
testing we are likely to make. 

Most of the positive responses (13 of 18) were vtry short, "Yes," or 
"Agree," or "Yes, this is an acceptable working hvpothesis." Here are two of 
the longer positive responses I found interesting: 

"I agree with the statement offered. I am compelled to note thr the 
tasks described in the statement are going to be quite difficult to complete. 
There will not be universal acceptance. There will be criticisms (legitimate 
and illegitimate) of the results for a long time to come and many will .an a 
course similar to criticisms of attempts to define and measure intelligence." 

satisfying reduction... (2) agreed here, (3) agreed here. Therefore, yes. And 
rather than get hung up on this question, I'd prefer to jump right in and see 
whether we can do it. If we can, terrific! If we can't, well, then the 
doubting T's will have a field day; but I'm prepared to take that risk..." 

One of the negative responses was expressed this way: "I'm, sorry, but I 
can't help but get hung up on 'operations' — the term so psychologizes and 
jargonizes... the question. Why not go for 'principles'?..." 

The strongest negative wa» registered by a person who said, "I fear this 
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proposal to define CT operationally may have the effect of ruling out, by 
definition, one major position on CT, a position with vhich many in the CT 
field may be in sympathy, at least to some degree." This person was concerned 
that defining CT as a list of skills would focused on veak-sense-CT and missed 
strong-sense-CT which relates to a person's character — "being self-critical, 
spoking to overcome blind spots, biases, prejudices..., [being] critics of 
one's society,... [seeking] what is of value in another's position..." The 
person asked, "Can these character traits be defined operationally?" 

In contrast, note this response: "Yes, [I agree with the statement] 
particularly if we confine ourselves to abilities and ignore dispositions..." 

Even though agreeing that "A subset of the processes that constitute CT 
can be assessed using the multiple-choice format that I assume is being 
sought," one person expressed serious concern saying, "I fear the creation of 
an instrument promising more than it can deliver — an instrument tovrted not 
for what it is, an assessment device measuring certain important, but 
rudimentary, CT activities, but rather as a valid and reliable assessment tool 
for CT." He maintained that "As is so commonly true when discussing 
assessment, the instrument and its characteristics would then dominate the 
social construction of what is being assessed, in this case CT. The initiation 
of CT activities, generation of appropriate CT strategies, and defense of a 
tentative reasoned judgment are. not susceptible to the type of assessment 
legislators are willing to finance or faculty are willing to undertake. The 
extensive writing or oral argument required to demonstrate CT, as I understand 
it, are not practical inclusions in an assessment instrument." 

Four who agreed with the approach mentioned important factors which relate 
to coMstruct validity. One mentioned the role of background knowledge in CT, 
another the role of divergent assumptions, another the relationship of CT 
skills to reading skills, and a fourth spoke to the need to validate any list 
of CT operations we might agree. There is much in the research literature 
about these problems. Steve Norris, in particular, has been working on 
strategies to respond to precisely these kinds of problems. 

Although I mentioned all the negatives, the positive responses were far 
more numerous than the negatives. A consensus to move ahead exists. But we 
must not forget the warnings and concerns of our colleagues. A great deal 
depends on what we come up with when we actually sit down to answer the 
question for Round 3^. since both the positives and the negatives were based on 
our ideas about what CT is. 

Rou . 2 question 3 asked if people would be willing to agree my proposal that 
to get started by giving priority to the college freshman/sophomore level. 

The responses ran: 14-yes, 3-no, 2-both, 3-abstain, and one that I could 
not figure out. Since the question was about priority and was not intended to 
exclude working at the K-12 level, which is, after all, what the APA Pre- 
college Committee is charged with doing, I believe we have sufficient consensus 
to focus initially on the lower division post-secondary level. 

Here are some "Yes, give priority to the frosh/soph level" comments: 
"Host if not all of the CT we teach is directed at this level" 
"I doubt philosophers should take the lead... when it comes to K-12." 
"I'm still unpersuaded to reverse my forauerlv expressed views — 
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Intro, undergraduate level first, until we get square there, 
where we live." 

"I feel rather strongly that ve should begin on the post-secondary 
level. X agree with those who say that that is where the 
strength o£ our membership lies and that that is where the vast 
majority of cur members teach." 

"Yes. That's where most of the pedagogical action is; that's where 
the students we're interested in testing and have relatively 
easy access to are to be found. Later, if we succeed at all 
here, we can extrapolate to other levels." 

"I agree in the light of what was said in Round 1." 

"Why not? It's what we know best, and ve can always move on to other 
levels later." 



On the negative side: 

"[Your] recommendation seems arbitrary and not consistent with the 
fact that this is a pre-college committee." 

"No, based on the name of the committee." 

"I would prefer to begin at the K~3 level. My position and 

opposition is based upon pedagogical and theoretical concerns 
which I assign a higher priority to than to political concerns 
related to the officers and membership of the APA.... I am 
willing to accept that the sub-committee begin with the college 
level but the project will have to be extended downward then..." 



Two people expressed the concern that this was a difficult question to 
answer until one knew the purposes for testing. [A point well taken.] 

Another suggested that our goal should be to make "contributions to the 
criteria for a college test... [but that] individuals should be encouraged to 
make up their own [assessment instiuments] and try them out, obtain 
correlations with other tests and with outcomes, and then subsequently compare 
notes with one another as to what worked and what didn't." 



Thanks to all who responded to Round 2. Your 
letters were most interesting and thoughtful. 
Don't forget Round 3 by June 25! 
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California State University, Fuilerton 



Fullonon, Cafjfornia 92634 



Department of Philosophy 
(714) 773-3611 



May 18, 1988 



Dear Colleagues, 

I hope this letter finds you well, happy and looking forward 



mentioned that I would be attending the "First National 
Conference on Assessing Thinking" in Baltimore and that I would 
be sending along a, report. The conference was very rewarding and 
the promised report is attached. 

I've also attached the current list of persons invited to 
respond to Round 3 in our Delphi process. We will have to freeze 
the list at this point. According to what I've read about the 
De^-phi method, once the central debate is joined, it can be 
disruptive to try to add people who have not the benefit of 
earlier rounds in the dialogue. With the circulation of the 
Round 3 question we have reached that point. 

Many thanks to those who have already sent there responses 
to Round 3. Don't panic if you haven't yet, though. The target 
date is June 25. 

Have a good summer. I look forward to hearing from you. 



to a restful as well a?^ productive summer. In my last letter I 




Sincerely , 



Pete Facione 
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Unlvenlty, Fullorton 

Fullerton, California 92634 



Department of Philosophy 
(714) 773-0611 



June 28, 1988 



Dear Colleagues, 



Thanks for the large number of interesting responses to the 
Round 3 of our critical thinking Delphi process. Judging from 
the length and sophistication of what many of you sent, it should 
take me a fair amount of time to organize and synthesize the 
material and then to frame fruitful questions for our next round. 

I expect to be working on this for several weeks, so if for 
some reason you haven't had , the opportunity to respond to Round 
3. please know that your ideas are more than welcome. 

Have a good summer and thanks again for so much high quality 
participation it's very encouraging. 

Sincerely , 




'.•1 

Pete Facione 
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California Stale University, Fullerlon 

Fullorton, Calilornia 92634 



DoparlineiU of Philosophy 
(714) 773-3611 



Sept. 1, 1988 



Dear Delphi Colleagues, 

I'm working through the 25 responses to Round i, which asked for your 
list of oore CT skills. The scope, quality and care evident in your responses 
is most impressive. Right now I'm onalyzins, synthes i x i ng , and organising the 
ideas so that I can play them Lack for your reaction and rectification. 
Expect the Round 4 letter in about two weeks. 

A Delphi operational rule I've adopted is to drop persons off the Delphi 
mailing IJst if they have made no contact whatsoever after receiving 
invitations to respond to at least three Delphi rounds. It's fair to say a 
person's sustained silence signals he or she wishes not to be included. 

Included here is somethinsj Phil Pecorino shared. It lists (a) Critical 
Reasoning and Informal Logic Texts, (b) Related Texts, (c) Logic Texts, <d) 
Newsletters and Journals related to CT, and (t) CT Centers and Organizations. 
Also, you might also want to write to James Bell, HowarcJ Community College, 
Columbia, Maryland, 210A4 and ask for a copy of his 132 page Guide to C T for 
Maryland Social Scientists. It includes a wealth of material on how CT is 
seen from different disciplinary backgrounds and it lists a number of ideas 
and resources. 

Thanks again for so much high quality participation in Round 3 ■■ it's 
very encouraging. You'll be hearing from me soon. 



Since ■• " ' " 




ERIC 



Pete Facione 
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California State University, Fullerton 



Fulierton, California 92634 



Department of Philosophy 



Sept. 23, 1988 



ROUND 1 



Dear Delphi Colleagues, 

Round 4 seeks verification of a list of CT skills and sub-skills. 
Please accept, reject, amend, and comment on the group's responses to 
round 3. Remember, the goal in this phase of our project is to arrive 
at an accord regarding the skills h& understand to be central to CT. 

Your responses to round 3 yielded 200+ pages. Some sent previous 
publications, some sent lists and commentaries, some wrote new pieces 
of clearly publishr»ble quality. .You tended to approach the question 
of identifying core CT skills and sub-skills four ways: (a) by appeal 
to your own experience and understanding, (b) by citation and comment 
on what other CT authorities (including others in our Delphi list) 
have written or said, (c) by describing the key characteristics of 
persons who have internalize CT, and (d) by consideration of what 
should reasonably be taught or included in a CT curriculum. In 
addition to differences of opinion, there were variations in 
disciplinary orientation, vocabulary, and emphasis. There were also 
differences in the specificity, depth, and scope of responses. Some 
were extremely general, others very specific. 

Distilling your opinions, positions, views, ideas, lists, 
descriptions, explanations, examples, counter-examples, caveats, . 
credos, and course outlines was one of the most intellectually 
interesting and stimulating experiences I've ever had the pleasure of 
attempting. Although I've been e iching and researching CT for two 
decades, I noticed that my own views on the range and character of CT 
expanded greatly as a result what you contributed in round 3. 

In naming and describing CT skills for Round 4, I intend to rely 
on standard English usage and to avoid technical or discipline- 
specific vocabulary. Your responses emphasized generic skills. 
Distilling your responses, I name and describe six generic CT skills 
and give two or three sub-skills under each. Clearly additions or 
deletions might be needed. To avoid prejudicing your responses at 
this crucial time, I do not indicate the numbers of persons who may 
have agreed on any given point. Areas of agreement and controversy 
will come out in round 4. In this round you are invited to make a 
number of kinds of responses to a variety of questions. However, 
because organising the material and framing the" issues was such a 
delicate and complex task, I urge you to read the whole package and 
get an overview of the terrain prior to starting to reply. 

Thanks again for the high quality responses to Round 3. To 
insure we are talking about the same things when we refer to CT 
skills, we'll need maximal participation in round 4. If you could 
consider this material and reply in 15-20 working days that would be 
wonderful. If you need more time, or want to discuss any aspect of 
this project, call me at CSUF (714) 773-3742, [office] or 372-3611 
[dept.], or 993-1356 [home]. 
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There are five parts to Round A\ 

<1) The listing of CT skills and sub-skills. 

(2) A model diagramming the flow of CT skill input and output. 

(3) A list of things some of you said CT is not. 
('1) A description of what is meant by "skill". 
(5) A list of caveats and comments you shared. 

Round 4j_ Part 1^ CI skil Is 

I 

Comments We've all seen examples, like the duck-rabbi t ,' of how comprehension 
combines impo.sinq an order on reality as well as discovering an order inherent 
in reality. Applying that lesson to the problem at hand, there are many ways 
CT skills and sub-skills could be organized. Even among those of you who 
essentially agree, the variations in your responses to round 3 illustrate 
this. After considering a large number of ways of organising things, I've 
decided on the configuration you will find below. True, the skills and sub- 
skills could have been named using other labels or grouped in other ways. To 
this add that some sub-skills may come into play in more than one general 
area, some can operate on the products of others, some presuppose others, some 
are almost always employed while others may not come into play except under 
special circumstances, and you have an even more conceptually complex 
situation. Now add that some of us mi ght. exc 1 ude one of the more generic 
groupings, to say nothing of how we might want to add, sub.tract, amend or 
rearrange the sub-skills, and the complexity of our task takes on greater 
magnitude. There were other ways to "see" the Round 3 data. So, beside 
asking yourself if each skill and sub-skill belongs in the list, and if 
anything central is missing, another question to ask is, like the duck-rabbit, 
can you see CT this way? 

Instructions: BeloN you wiii TiftJ S'list naaing and describirig su core CT 
skills and sub-sk ills . flead and consider the entire list* Then, 

(1) Consi der i nq each s k 1 1 ! and sub-sk til , marl, ea^ch, "Yes" i± jjou, v er i f y 
LL to. be a. cor^e. CT skill or. sub-ski 1 1 and "No" i_f. i£u woul d argue to. exclude 
i t . St ate your reason f or excludi nq an^ marked "No" . 

, (2) (lake nee ded substantive amend m ents to the descr i pt ions of any sub- 
skill , and. e.".pl ai n why the change i s needed . Add any missing s ub-skill, name 
§1} J. d escr i be i t . 1 ocate 1 1 within one of. the siji C± s I; i 1 1 s . Con si deri ng the 
su b-sk i lls w i t hin each of. ihe, sjji CJ_ s kills, indi cate which sub-ski lis. ij_ 
iMa. SilPiUJ. ki. moved, to. some ot]ier, skill. 

(U_ Amen d the l_ij.t. of. 5i.x. CT. skills, l± delet e a. skill, indicate 
illL^i. to. 1°. with i ts s u b - 5 1. i 1 1 s . I f y ou add a. missing CT skill, name, and 
desc r i be i t . name and describe i_t_5. s u b - s k i II s . explain how i t i s central to 

how i.t, differs f rom an^ of. the six s k, i II s on the current list, and whv i t 
m eri ts being listed at. the. 1 evel p_f. a. skill rather than a sub-ski 1 1 . 

li.L Consi der the NOTE reqardi nq the rol e and composi tion of. a. human's 
i:n oNled<}e b ase . verify, amend . commen t on any aspect of this. 

( 5) Make a ny needed edi tori al changes . 

LkL Co mment , i f you va s h . on the ent i r e l ist of. s k, i 1 1 s . i ts organ i z at i on . 
u tility, what have you. 

1 urge you to read the entire list of 
skills and sub-skills before beginning to 
respond to any specific item. Thanks! 
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PROPOSED'. Core Critical Thinking Skills 

TAXONOMY 

1. Interpreting — Observing, Decoding, Clarifying. 

2. Inforrinq — Querying, Con je?c tur i ng , Drawinq Corjcl usi ons. 

3. Analyninq — Locating A.-guments , P,^r5ing Arguments. 

4. Eve-duating — Veri-fying Claims, Assessing Logical Strength 

5. tixpressi ng — Stating Results, Describing Proceciures. 

6. Nonitoring — Regulating, Reviewing, Correcting. 

DESCRIPTIONS OF SKILLS AND SUB-SKILLS 

YES/NQ 



1. INTERPRETING: To comprehend the significarice of a wide variety of 
experiences, situations, judgments, beliefs, rules, procedures and 
cri ter i a. 

1.1 OBSERVING! To detect, attend and correctly perceive 
experiential input with particular focus on input that conveys or 
is intended to convey data, information, or inferential 

rel ati onshi ps. 

1.2 DECODING: To detect, attend to and rorrectly perceive 
the informational content, rules, procedures, criteria, and 
inferential relationships expressed in various convention-based 
communication systems, such as language, social behaviors, 
drawings, numbers, signs and symbols. 

1.3 CLARIFYING: Ta make explicit, through stipulation or 
description, the contextual, conventional and/or intended 
meanings of words, ideas, concepts, statements, behaviors, 
drawings, numbers, signs or symbols; to remove confusing . 
vagueness and ambiguity; to facilitate communication. 



2: INFERRING: To secure eleannts needed to aake inferences and to 
deteroine the inferential relationships betueeri or floning froa 
statements, descriptions or representations. 

2.1 QUERYING: At any. point in the CT process, to recognise 
the need for evidence or information of some hind, and to 
formulate and execute a strategy for seeking and gathering that 
evidence or information. 

2.2 CONJECTURING: To formulate alternatives, to develop 
hypotheses, to postulate suppositions. 

2.3 DRAWING CONCLUSIONS: Given a set of statements, 
descriptions or representations, to educe their inferential 
relationships and to educe the consequences which they 
support, warrant, imply or entail. 
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NOTE.; As many of you argued, all CT skills, but particularly sub- 
skills like 2,1, 2.7 and 2,3 presume a knowledge-base , A human's 
knowledge-base composed of at least these things: 

(a) a world view which includes one's understandings of 

* what is real (a metaphysics), 

* how knowledge is gained and refined (an epi stemol ogy ) , 
« what is important or valuable (a value theory); 

(b) »a data base , including one's opinions? beliefs, 

experiences, etc. as filtered through the world view; 

(c) an i nf erence eng i ne which includes 

* general rules for drawing logical inferences (a logic), 

* sets of procedures and criteria appropriate for making 
reasonable judgments within specific areas of human 
thought and inquiry (discipline-specific ruleSt) 

Executing sub-skills 2.1, 2.2, and 2.3 as well as 4.1 and 
4.2, -more effectively can be achieved by learning how to think 
logically, by expanding one's repertoire of sets of procedures and 
criteria used in different areas of human thought and inquiry, 
and increasing one's base of relevant data. 

An implication of this analysis of CT skills and sub-skills 
is that they transcend specific disciplines, but executing them 
demands background knowledge, some of which is specific to how one 
goes about making reasonable judgments in different realms. 

Becoming adept at CT involves learning CT skills and 
learning to use those CT skills more effectively in different 
conte,^:t5 hence the importance of a liberal education to go 
along with one's CT ability. 

3. ANALYZING: To ideutify the inferential relationships between 
statements^ descriptions or representations which express experiences, 
situations, judgments, beliefs, or opinions. 

3.1 LOCATING ARGUMENTS: Given a set of statements, 
descriptions or representations, to determine whether it does 
express or was intended to express a reason or reasons in support of 
some claim, opinion or point of view. 

3.2 PARSING ARGUMENTS: Given a the e;:pression of a reason 
or reasons in support of some claim, opinion or point of view, to 
identify: (a) the intended conclusion, (b) the premises and 
reasons advanced in support of that conclusion, (c) additional 
unexpressed elements of that reasoning, such as intermediary 
conclusions, unstated assumptions, and id) for exclusion, any 
items contained in body of expressions being parsed which are not 
intend to be taken as crucial to the reasoning being expressed. 



4. EVALUATING: Jo assess the cr ed i b 1 1 i iy of statements, descriptions or 
representations; and to asse.^s the strength of the expressed inferential 
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re lat ioTi shi betp^een such statements, descriptions or representations. 

4.1 VERIFYING CLAIMS: To assess the degree of confidence 
to place in a given statement, description or representation, 

4.2 ASSESSING LOGICAL STRENGTH: To determine the nature 
and quality of expressed inferential relationships; to judge 
whether the assumed truth oi the premises of a given argument 
justify one's accepting as true, or very probably true, the 
ej:pressed conclusion of that argument. 

5: EXPRESSING: 7li siatef describe or represent to one's splf or others 
the results of one's CT actii'ities and the way one uent about producing 
those results* 

5.1 STATING RESULTS: To produce accurate statements, 
descriptions or representations of the results of one's CT 
activities so as to analyse, evaluate, infer from, monitor or 
remember those results, or so as to communicate them effectively to 
others. 

5.2 DESCRIBING PROCEDURES: To produce accurate statements, 
descriptions or representations of how one applied and executed 
any CT skill or sub-skill so as to evaluate or monitor one's 
proficiency, or so as to communicate to others about how one went 
performing a given CT skill or sub-skill. 

6: MONITORING; To regulate diJ aspects of one's ONn CT acti^ities^ the 
elements used in those act iv it ies p and the results produced by those 
acti^^ities, particularly by applying the skills of ana 1 yz i ng , and 
evaluating to one's own inferring with a view toward confirming, 
validating and/or correcting the results. 

6.1 REGULATING: To sequence one's execution of CT skills 
and sub*ski lis. 

6.2 REVIEWING: To examine one's own CT activities and verify 
both the results produced and the correct application and eMecution 
of each CT sk'li and sub-skill involved, 

6.3 CORRECTING: Where errors are found in one's own CT 
activities, to co.'rect those errors and remedy their causes. 
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Round 4j_ Part 2. An Input/OuLput Model of CT Ski lis 

Comment: There'E no dcnyiny that tfie mode?! I've coiuo up is l\t?ctvily 
in-nue?nced, for good or ill, by my research in cromputGi- science «i\nd 
arti -f i ci a-J. i ntel 1 i gf^nce. The c^rrows indicate the direction of the 
•flaw o-f input and output as it circulates btstween and is operated upon 
by the CT cskills, here "conceived of as functions. 

In this model, (-iMprgssi no r6?ceives output in the -fortn of 
information • from the knov;ledge base, the results of other CT 
functions, and lists of the steps pcsrformed during other CT functions. 
l-HpreiiBinrj LlifMi forniaLs thai, output for use by t\)a ne;!t CT f uf icl: i on , 
for storage in l:he knowledge-base, or for transmission out of the CI 
cycl e. 

Within the CT cycle, monit or i ng receives material from express! nq 
and, as the regulatincj and sel f -correct ing function, determines uhere 
it should go ne;:t. -It can route things into or out of any of the five 
other CT skills or the know! edqe-base. Kor e::ample, rnonitorinq can 
loop material back through any skill. Thus, it can send the results 
of one's own in f err ing for review by routing them to eval uati nq before 
allowing ejiere_ssi.nc;j. to transmit them to others. Or, it can route 
inftarmation from the knowledge bas^i to e va Ku^l^i n^^ to help it verify a 
claim, or to il}±erj-ij^{i to help it draw a conclusion using criteria 
ispecific to a given discipline. 

The four CT s^:ills many of us cvpt'nd so much time helping situdtMits-:. 
beconu? praficic;nt al. , namely ijAtl-LQPriit.^..ay..v :^^J:L^l..yjliDJ4 f?Vj;il uat i riq and 
ijl;.tt^?r.ril")9 receive matt?rial routed to them by the (iionitorinq function. 
They operate an this material. And tfien thc^y send the results to the? 
expressing function to be formatted fur delivery elsewhere. 'Ihese 
skills also output a record of the steps they performed in coming to 
those results. This record is crucial if the monitoring function is 
to work correctly, since it must check not only what was achieved but 
how it was achieved. 

The .f!jnpy..^lif-!cje base, is a storehouse from which interpreting, 
anal y:;: ing, evaluating and inferring drav^j resources. it stares the 
output of any CT function, when directed to do so by ttie monitoring 
function. The k n^i'l. l-S/^illi?:. b_a sjs also f i J. t e r s r a i-j external input and, 
thus, influences the i n t er pr et i nq of what we are observing, decoding 
or clarifying. 

Instructions: Cpf^ai der the model Dn. the next. tLag_e_._ 

P oe s i t m_a}:;.e. sj-;ns.e ta_ ysil-lZ. 
i-t. !5fI!^kU:lf\tJ^.2 
•rL'. i.s. it. u.^.?i.uiJZ.«_ 

'}.:, 5?i..r'S.?r. C'slCli^' pt-^PrpJ.^. i.lDSi J-li_ctor_i.a_l_ mqd_el_E_ [iel_p+Lil i_n 

unclC-C s.tancJ.]^ CQ.'.'.'P..l..?.fl L ej/;it i_qnsjjjix^^ cain ygu suggest how tin.is one 
'l'i..y,b.:t. be 1 ll'fjrpyed prj,. i,f v:.P.u r ^.i^iojum e n d cJj_t car_d_i ng i_t_.v can. y^gu prpp.os.e 
AD. pA;l.t.erjnat_i^ 
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Round 4i. Part 3^. What CT Not 



Comment: Many of yrju di st i nqui shed CT skills •from athe?r closfi-Iy 
related things*. B6?law i «> <^ list of what various persons aaid was 
not. CT. Just because somethinq is on this list does not mean a 
p«rs»t3*n aoes not use CT in doing the? thing nor thac a person rniqht 
not do the thing before, after or during CT. 

Instructions: PLtase co.nsi_d.er. this 1 i st , verify ,i.t_,. anierid i t . and 
c:om(nt>:nt on ij;.s ut i :i i tv . " 



;l. Sensing, (Seeing ^ Touching, Hearing, etc.) 

2. Reading, Listening, 

3. Speaking cr writing, 

4. Motivating, persuading, selling, 

5. Interrogating, cross-examining, petitioning, 
ij. Phy-sjically investigating the world around, 

7. Troubl e-i-jhout ing , probl e«i-sol vi nq , puzrle solving, 

S. Decision-making, selecting, choosing, e::erci s.\ nt) one's will, 

9. Planning, defining goals and objectives, 

10. Finding or ascribing a meaning to art, life^.or events, 

11. Defending an opinion or belief, arguing a case, 

12. Managing, administrating, cr governing persons or things, 

13. Philosophizing, 

14. Conducting research within any particular discipline, 

15. Experiencing, fe'-jlinq, ernotinq, or empathizing, 

16. Communicating using language. 



CRITICAL THINKING IS NOT 



YES/NQ 
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Round 4j_ Part 4j_ The. Concep t g± a Ski 1 1 

Comment: It became clear from your responses that it would be useful to have 
an understanding about what a skill is. Although there may be very little 
disagreement about this, s'ome of you mentioned subtleties others may or may 
not accept. Based on your contributions I've written a little narrative. 

Instructions: Revi se . edi t and coinpl ete the foil owino narr ati ve; 

A skill the ability to do saaethiug weii. Having a skill includes knoNing 
nhat to dOf Nheti to dc it, and bon to do it. That is, being skilled at 
soaething involves knomng a set of procedures , jud\^'iug Nhen to apply those 
procedures, and being proficient or adept at executing those procedures. 
Skills can I iught in a variety of nays including, but not liaited to, 
coaching, deaon-^t r at ing , and training. Part of the teaching involves aaking 
the procedures explicit and shoNing nhen and hoN they are enployed. Skills 
can be learned through a coabinat ion of observation, guided practice, drill, 
and self -cor r 3ct 1' n . Persons can be judged as being tore or less prof icient 
in a given skill. The first nay of assessing is to observs the skill as it jf 
bein<^ perforaed. A second nay is to compare the outcoaes lif any) that result 
froa executing a given skill against sose set of criteria. A third way is to 
query persons and receive their descr ipt ions of the proced:ires and .judgitents 
they are using as they perfori that skill, wouid use if they were to perform 
that skill, or did use nhen they perforaed that skill. Sj>ce wi? cannot 
directly observe the perforaance of eental skills the way we can physical- 
skills, only the second and third nays are available to those Nho uould assess 

CT. The sec(}nd nay is lis not) superior to the third because , 

provided that 
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Round 4l Part 5t Caveats and Cautio ns 



Comment A number of you sent comments. Some were intended as caveats or 
cautions, other as encouragement. I appreciate them all. I thought some 
might be good to share; In many cases to save space I have paraphrased. You 
fltight agree or disagree with what your colleagues have said. Or, reading this 
'list might prompt you to pass along a contrary view. Let's find out. 



Instructions Consi der these comments. Make my remarks you wi sh , whether ui 
agreement gr_ di saqreement . Use these remarks a5> spring boards for your own 
ref 1 ec ti ons. 



AGREE/DISAGREE 



1. Keep in mind that our goal in defining CT is to do 
some assessment. But you don't just start testing people. 
As!:essment needs focus and purpose. 

2. I agree with working at the college level only because 
I want to have an idea about what we should be doing with kids 
in K-12. I think we can use what we say about CT for college 
frosh/soph to guide curriculum development in CT. Of 
course, we will have to adjust reading levels, background 
knowledge expectations, and lots of other things. 

3. Skills are not the same as operations. Resist 
behaviorism! Behaviors give evidence that a person has a 
skill or ability, but a skill is not a set of behaviors. 

4. Don't trap yourself into using the jargon or 
vocabulary of any one discipline (especially philosophy) when 
you describe CT. 

5. Although no word will escape criticisio, don't use 
"deduction" or "induction." Avai d semantic spats. 

6. If CT is a set of attitudes as well as a set of skills 
that poses no problem for assessment because we can just 
develop ways to assess the CT attitudes, too. 

7. CT is thinking skills. Saying CT is a set of 
attitudes may be a way of describing what people who are good 
at CT arR like, but it is not a way of describing critical 

th king itself. 

8. Even ii we agree on w^it CT is, we still have to face 
the problem that any student might get the right answer on a 
CT test but for the wrong reason, or might get a problem wrong 
but have done a good job of CT« 

9. When assessing CT we should not duplicate efforts with 
areas already well tested by existing instruments, such as 
covered by reading or intelligence tests. 

10. You don't have to test -^very ingle CT sub-skill to 
decide that f: person i good at CT. 

11. I looked at that list of e:<pe' vj and you have all the 
big names I can think of, but you can't possibly e>:pect those 
people to agree. If they did agree, even on what CT is, that 
would really be something. Good luck! 



FEEL FREE TO DISCUSS THIS MATERIAL WITH COLLEAGUES, IF YOU WISH. 




PLEASE DCN'T REPRESENT ANY OF THIS LETTER AS THE OPINION OF OUR DELPHI GROUP. 
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California S\aU University, Fullerton 

Fullerton, California 92634 



Department of Philosophy 
(714) 773-3611 

Dear Delphi Colleagues, 



Nov. 22, 1988 



During this Thanksgiving season and I want to express my gratitude to 
you for your generous participation in this research during 1988. The many 
responses to the very long and di£:> icult ROUND 4 have been most gratifying. 
There won't be any more "ugly-long" rounds like that, I promise. 

Let me pass along this quick review of what we accomplished in 1988. 
First, working under the auspices of the American Philosophical Association 
Committee on Pre-College Instruction in Philosophy, we built the Delphi 
list of experts. By your recommendations, during Rounds 1 and 2 (Feb. 11 & 
Mar. 14) we expanded the original APA list of about ten names to a working 
group which numbers around 45 active participants and which, I am proud, to 
say, includes many of the most important people in CT research today. 

During Round 1 and 2 we agreed that whatever CT is, we would be 
possible for us to make the concept operational to the extent that 
important parts of CT could be assessed validly and reliably. We also 
agreed to begin by identifying the core elements of CT expected at the 
Frosh/Soph. general educfition college level. We agreed to use this college 
level theoretical-construct of CT to guide what is said about CT the K-12 
levels. Round 4 (Sept. 23) sought to verify the concept of CT which 
emerged from Round 3 (May 4) . A quick look at the results of Round 4 is 
roost encouraging! 

Along the way we shared journal articles, lists of existing CT tests, 
CT bibliographies and o«;her items of mutual interest. The Delphi, however 
is not a substitute for the fine work being done by journals, newsletters 
and the many centers for CT that have emerged in recently. 

While I work on ar.alyzing the results of Round 4, I invite you to 
consider where do we go from here. Last spring I outlined a four phase 
Delphi project. When ve achieve consensus on the core list of CT skills 
expected at the lover division college level, we will have completed the 
two of the four phases. Originally phases 3 and 4 were described this way: 

Phase 3: Recommendations 

The goal of this phase is to communicate our findings about what 
CT is and whether there is an adequate way of characterizing CT 
operationally so as to permit its being tested at some 
educational level. Depending on our results in Phase 2, we will 
recommend either that programs aimed at testing CT be abandoned, 
or that they be focused in certain ways. If this is the 
direction Phase 3 takes, then we will also try to come to 
consensus on recommendations regarding the relative importance of 
different kinds of CT sub-skills and possible strategies for 
accessing and measuring those sub-skills. 

Phase 4: Design and Validation of Model Testing Strategies 

Contingent on the results of earlier phases, the goal, if it were 
considered achievable in principle, would be to construct and 
evaluate: different approaches to testing CT at some appropriate 
educational level or levels. We might find ourselves breaking 
^ ER?C in' o sub-committees to achieve this goal/ although all hqx± will o^^ 
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have to be guided by ths agreements reached in earlier phases and 
as well as by the special expertise of those viho understand the 
intricacies of designing, piloting, norming and validating 
educational tests at 'specific educational levels. 

To date I've done little regarding preparing to communicate our 
findings. Two CT newsletters intei'ested in publishing something regarding 
our results, have contacted me. Also, the Pre-College Committee scheduled a 
session at the March 1989 APA meetings at UC Berkeley. At that session 
I'll be outlining our Delphi process and what we have agreed on by tnat 
time. More suggestions are most welcome. 

For many reasons I am extremely skeptical about actually developing a 
good CT test using a Delphi process. Once we declare consensus regarding 
the theoretical construct of CT for the general education (lower division) 
college level our choices include at least these three, and maybe more. 

(1) Ve could move on to consider questions like these: Given what we 
understand CT to be at the college level, what does CT m^an a"- different 
grade levels in K-12? What is the relative importance of the; skills or 
sub-skills in our college-level CT construct in terms of testing, say 
junior high school students? How might one write a question which assesses 
a given sub-skill in, say 5th graders? 

(2) Having declared consensus on a conceptualization of CT for use at 
the college level, we might recommend that test makers at all educational 
levels be guided by our conceptualization. But we, ourselves, might decide 
to leave the matter of writing specific tests for specific age groups to 
others, better qualified than ourselves for developing and valida«-ing such 
instruments. Those of us interested in specific grade levels could be put 
in contact with one another. 

(3) When we reach consensus on the CT concept as it applies at the 
college level, we might recommend examples of how questions framed 

to address these skills and sub-skills in college students or K-12 
students. We could share these exair.ple questions and evaluate them. Those 
which we think a priori might be good to assess certain skills or sub- 
skills, could be included in recommendations we make regarding CT 
assessment. These questions would not be a CT assessment tool. At best we 
might think of them as models of how to crnceive of questions that might be 
included in a CT assessment. 



Note: Even with questions which a priori seem to address the proper 
concept of CT and avoid other difficulties (like relying on special 
background knowledge or esoteric vocabulary) , there is still the problem of 
a posteriori verification. Steve Uorris has done important work on how to 
overcome the "construct-validation" problem, namely determining, for any 
given test item, if students get right answers because of good CT skills 
and wrong answers because of inferior CT skills. 

Give the issues of what do to next some thought. I would welcome 
hearing from you on this. Have a joyous holiday season and thanks again 
for contributing so generously during phases 1 and 2 of this project. 



Sincerely, 
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California State University, Fullerton 

Fuilorton, Calilornia 92034 



Department of Philosophy 
(714) 773-3611 



Feb- 7, 19B9 

Dear Heroic Delphi Colleaguesi 

34 people responded to Round 4. (\ great return, our biggest so ^ar 
both in nuiDbers of persons and in numbers of pagesJ 1 will tally, analyze, 
distill and share what people 5^aid on each chun!: of Round 4. lhat way you 
will have the benefit oi an overview oi what others in the Delphi are 
thinking. rHit, intending to spare all of us any more horrendously 
burdeosonie rounds, when it comes to formulating the next set of questionsi 
1*11 keep thr^m as short and focused as possible* 



\ particularly want to thank the rrany people who sent me detailed, 
thoughtful, (and even footnoted) rvssponses. Several ran near ten pages 
single spaced. Not intending to diminish the value of the brief "Yes/No" 
responses as we approach consensus on crucial points, I must acknowledge 
and convey my appreciation for the many extra hours of work several of you 
are putting in. Also, I've learned a great deal from your sensitive and 
sensible comnionts. 



Yes, we are approaching consensus. My first two readinqs of the input 
on Round 4 is that we have a great deal of accord on the list of CT skillf), 
although there will be scores of adjustments and amendments to be made. 
Also, your comments on what a skill is and hoN a cognitive skill can be 
assessed suggest we arc close t)\ere too. But I'll be summar.i zi ng all that 
and more for you very soon. 

In the next few weeks I'll be sending you a few quick short rounds. 
My plan is to focus each brief letter on one discrete aspect of our work. 
He should try to complete several mini--rounds this spring. That way I'll 
be able to rough out a first draft of our report to the APA Pre-College 
Philosophy Committee during the summer, be thinking about recommendations. 

I've ericioseU a list of the people who are par tici pal inq in our Delphi 
research project. The "Rl" •*R2" and "R4" symbols indicate the rounds 

to which the person has wontributed. The "J" symbol means the person 
communicated an interest to be involved, but has not responded yet to any 
rounds. Currently there are 56 names on the list uf people being invited 
to respond* A few have never responded in any way. So, when it comes to 
making our final report, I expect there to be around 50 of us m the group. 

''11 be in touch with a summary of Round 4 and some mini-rounds 5, 6 
etc. very soon. Thanks again for all your work. 



Si ncer el V 




Peter A. Faci one 
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California State University, Fullerton 

Fullerton, California 92634 



Department of Philosophy P^^^*^^ §^ 

(714) 773-3611 l^eb. 28, 1989 

Dear Delphi Colleagues, 

One of you wrote, "I'm beginning to think that to have done all that Cwe 
have doneJ and not have tried to define CT may turn out to be a mistake." 

Also in response to ROUND 4 three or four of you commented like this: "I 
have no major quarrel with any parts of your organisation, CbutD your 
emphasis leaves out a major component of CT — the dispositional 
component and the set of values inherent in being a critical thinker... I 
think it is a good working model of CT skills, but an incomplete picture 
of being a critical thinker." 

In view of the many positive responses to ROUND 4, yet sensitive to the 
concerns raised by comments like the above, ROUND 5 begins by building on 
our success in articulating a decent first draft list of CT skills. In 
this letter we start right in on the question: For purposes of aer ral 
education assessioent at the college iower division leuel, Nhat do 
experts recommend be included as a core critical thinking skill? 

In addition to asking your endorsement of a revised draft of the skills 
dimension of CT, this letter also shares some key ROUND 4 results and 
some of the many useful comments you sent. 

The next letter, ROUND 5-B, works on the two other aspects of CT you 
commented on in your ROUND 4 responses — namely CT's dispositional and 
the normative dimensions. Some of this also finds its way into the 
revised skills statement — see 6.1! ROL, D 5-C picks up the remaining 
pieces of ROUND 4 and asks your approval of an outline of our report to 
the AFA Committee on Pre-College Philosophy. 

The table at the end of this letter shows that over 85'/. of us, (23 of 
26), could lie described as fundamentally in accord with our first listing 
of CT skills. The second draft you are now being asked to consider 
endorsing was prepared in view of t\\s many helpful comments and 
suggestions you sent in. I am very optimistic about the revised 
statement of CT skills, first because ROUND 4'?; draft was approved by 
such a solid plurality, and second because your suggestions helped me 
substantially strengthen and enrich that statement. 

Since we are very close on so many things, your approval or disapproval 
of the expressions of our views presented in ROUND 5 should clarify 
things enough for me to start putting together our report to the APA 
Committee on Pre-College Philosophy. Where v-^e have consensus our report 
will say so. Where we diverge, it will say that as well. 

I truly appreciate all that you have already contributed, and I realize 
you you are all very busy folks. Yet I beg your continued indulgence. 
Please respond to the three round 5 letters with all reasonable dispatch. 
All responses are welcome, no matter how brief or selective. 



With sincerest gratitude, 
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Half of this letter shares comments regarding defining and testing CT 
made in response to ROUND 4, Before digging into our revised statement 
of CT skills, you might jump to page 6 and look through the comments or 
examine the tabular results of our earlier Nork on page 10, I learned, 
for example, that not listed in our original statement was a skill the 
majority feels is part of CT arguing. And, given what this Delphi 
project is all about, how on earth could I have omitted the CT skill 
analyzing an idea from the first draft? 

ROUND 5-A, PART 1, IZ. SKILLS 

INSTRUCTIONS: Consider the following anended description of CT skills along 
with the accompanying statements. Starting with the title and preamble, make 
any needed changes, deletions, or addiUons (editorial or substantive). After 
working through the descriptions and statements you will be asked specific 
questions regarding endorsement. Please respond to those question as well« 

Skills Dimensions of Cr i tical Thi nki ng 

For purposes of general education assessoent at the college, lower 
division level, we understand CT to include the cognitive skills of 
interpreting, inferring, analyzing, evaluating, expressing and monitoring. 
Because of our collective conviction regarding their centrality to CT , we urge 
those persons interested, in assessing the skills dimensions of CT focus on 
these six abilities. However, since CT can be subcl assi f i ed in a number of 
legitimate ways, our subcl assi fication should not be interpreted as an 
educational taxonomy nor as implying or presupposing any psychological, 
logical or epi stenol ogi cal order or sequence of skills. While including those 
skills we take to be central to CT, we do not claim' that our list is 
exhaustive in either breath or detail. 

Critical thinking involves actively interpreting one's experiences and 
self-consciously making and expressing one's analytical^ evaluative and 
inferential judgments regarding what to believe or do. As such, critical 
thinking is a pervasive and mul ti -di mensi oual huoan phenomenon involving both 
dispositions and skills. Without diminishing the vital importance of 
cultivating CT dispositions throughout the K*-12 and post-secondary educational 
process, we have here chosen to focus our attention on listing and describing 
CT abilities. As a goal statement of what a generally educated college level 
critical thinker should be able to do, we hope our consensus description of CT 
skills will assist in CT assessment and CT curriculum development both at the 
college and the K-12 levels. 

Among the nany ways one might improve one's CT are by reflecting on one's 
reasoning processes and learning how to think more analytically, objectively 
and logically, by expanding one's repertoire of those more specialized 
procedures and criteria used in different areas of human thought and inquiry, 
and by increasing one's base of information and experience. An implication of 
our analysis of CT skills, however, is that CT skills per se transcend 
specific disciplines, yet executing them successfully in certain contexts 
demands background Knowledge, some of which may be specific to how one makes 
reasonable judgnents in that context. Since becoming adept at CT involves 
learning to use CT skills effectively in many different contexts we cannot 
overemphasize the value of a solid liberal education to supplement the honing 
of one's CT skills and the cultivating of onf^;*| CT dispositions. 
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Waies of Core C7 SkiUs aud Sub-^^kilU 



\. Interpreting 

2. Analyzing 

3. Evaluating 

4. Inferring 

5. Expressing 

6. Honi toring 



CategoriEing, Investigating, Decoding , CI ari f ying. 
Analyzing Ideas, Identifying ArgumentSi Analyzing Arguments. 
Assessing Clair^^i Assessing Argunents. 
Querying, Conjecturing, Concluding, Developing Reasons. 
Stating Results^ Describing Procedures, Stating Arguaients. 
S£l f -Bxacii nation, Se If -correction. 



Descriptions of Core C7 Skills and SuL^skiUs 

\. INTERPRETING: To comprehend the meaning or explain the significance of a ^ 
wide variety of fsxper i enties, situations, data, events, judgments, conventions, 
beliefs, rules, procedures and criteria. 

1.1 CATE60R1ZINB: To formulate categories, distinctions, frameworks or 
questions, end to describe experiences, situations, beliefs, events, etc., so 
that they take on comprehensible significance or fneaning, as for example to 
recognize a problem and define its cnaracter without prejudice to inquiry. 

1.2 INVESTIBATIN6: To actively seek, attend to, discriminate and describe 
experiential input relevant to a given situation, problem or concern; to 
gather input that conveys or is intended to convey data, information, or 
inferential relationships, as for example to gather evidence relevant to 
solving a problen in the light of how that problem is defined. 

l.Z DECODING: To actively detect, attend to and correctly understand, 
the informational content, 2ffective purport, directive functions, intentions, 
purposes, symbolic significance, values, views, rules, procedures, criteria, 
or inferential relationships expressed by others in conventions-based 
communication systems, such as in language, social behaviors, :*'u.^ings, 
numbers, signs and symbols. 

1,4 CLARIFYING; To explain, paraphrase or make explicit, through 
stipulation, description, analogy or figurative expression, the contextual, 
conventional or intended meanings of words, ideas, concepts, statements, 
behaviors, drawings, numbers, signs, symbols, rules, events or ceremonies; to 
an extent proportionate with the purposes at hand, to use stipulation, 
description, analogy or figurative expression to remove confusing, unintended 
vagueness and ambiguity, or to design a reasonable ^'ro::edure for so doing. 



2. ANALYZING: To identify the intended inferential relationships among 
statements, questions, concepts, descriptions or other forms of representation 
intended to express beliefs, judgments, exper i Cinces , information, opinions. 

2.1 ANALYZING IDEAS: to identify expressions used in communication and 
determine the role they are intended to play in arguing or persuasion, as for 
example to identify a phrase intended to trigger a sympathetic emotional 
response and induce an audience to agree with an opinion; to identify related 
judgments, views, or concepts and to deternine the conceptual similarities and 
differences between them; to identify issues or problems, determine their 
component parts, and identify the conceptual relationships of those parts to 
each other and to the whole. 

2.2 IDENTIFYING ARGUHENTG: Given a set of statements, descriptions, 
quv^stions or representations, to determine whether it does express or was 
intended to express a reason or reasons in support of or contesting some 
claim, opinion or point of view. 

2.3 ANALYZING ARGUMENTS: Given the expression of a reason or reasons 
intended to support or contest some claim, opinion or point of vien, to 
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identify: (a; the intended main conclusion, (b) the premises and reasons 
advanced in support of the roain conclusion, (c) further premises and reasons 
advanced as backup or support for those premises and reasons intendiid as 
supporting the nain conclusion, (d) additional unexpressed elements of that 
reasoning, such i ntcrfsedi ary ccncl uzi ens , unstated assucpt i on^ or 
presuppositions, (e) the overall structure of the argument or intt^nded chain 
of reasoning, and (f) any items contained in body of expressions being 
examined which are not intend tc be taken as part of the reasoning being 
expressed or its intended background. 

3. EVALUATING: To assess the credibility of statements, descriptions, 
questions or other representations expressing experiences, situations, 
beliefs, judgments, or opinions; and to assess the strength of the expressed 
inferential relationships among such statements, descriptions, questions or 
other forms of representation. 

3.1 ASSESSING CLAIHS: To assess the degree of credibility to ascribe to 
a source of information or opinion; to a. ,> the relevance of questions, 
information, principles, rules or procedL"^vi i/ections to a given issue or 
concern; to assess the truth or the level j idence to place in any given 
representation of an experi ence, si tuati on , judgment, belief or opinion. 

3.2 ASSESSING ARGUMENTS: To deteraine the nature and quality of 
expressed inferential relationships; to judge whether the assumed truth of the 
premises of a given argument justify one's accepting as true, or very probably 
true, the exprr?ssed conclusion of that argument; to anticipate and raise 
questions and objections, and then to assess whether these point to 
significant weakness in the argument being evaluated; to determine whether an 
argument relies on false or doubtful assumptions or presuppositions to judge 
how crucially these affect its strength; to judge between reasonable and 
fallacious inferences; to judge the probative strength of an argument's 
premises and assumptions with a view toward deterviining the acceptability of 
the argu/Bent; to determine and judge the probative strength an argument's 
intended and unintended implications with a view toward judging the 
acceptability of the argument; to judge the extent to nhich additional 
information would strengthen or weaken an argument. 

4: INFERRING: To identify and secure elements needed to make inferences and to 
determine the inferential relationships between or flowing from statements, 
descriptions, questions, or other forns of representation on the basis of 
which inferences can be drawn. 

4.1 QUERYING: to recognize the need for evidence or information of some 
kind, in particular to recognize which statements, including those offered as 
premises, need justification, and to formulate and execute a reasonable 
strategy for seeking and gathering that evidence or inforraation. 

4.2 CONJECTURING; Given a problem, question or point of view on an 
issue, to formulate multiple alternatives, develop hypotheses, or postulate 
suppositions, and to design reasonable strategies for determining their 
plausibility, viability or rt^lative merit; to objectively draw out the 
presuppositions and the consequences of decisions, positions, beliefs or views 
with which one might agree or disagree. 

4.3 CONCLUDING: Given a set of statements, descriptions, questions or 
other forms of representation, to educe with the proper level of logical 
strength, their inferential relationships, both deductive and inductive, to 
educe the consequences or the presuppositions which they support, warrant, 
imply or entail; to successfully employ par ous sub-^species of inductive or 
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deductive reasjoning, as for example to reason analogically! arithmetically! 
di al ect i cal 1 y , scientifically^ etc. 

4,4 DEVELOPING REASONS: Given a question to be answered or a position on 
an issue! use appropriate inductive or deductive modes of inference to 
articulable reasons for answering the question one way as opposed to another! 
or for supporting or for opposing the position. 

5: EXPRESSING: To state! describe or represent to one'L self or to others the 
results of one's reasoning and the way ons went about producing those results. 

5.1 STATING RESULTS: To produce accurate statements! descriptions or 
representations of the results of one's reasoning activities so as to analyze! 
evaluate, infer from! or monitor those results! o'^ so as to accurately and 
effectively recall or represent those results to one's self or to others, 

5.2 DESCRIBING PROCEDURES: To represent as cUarly as possible how one 
came to one's interpretations! analyses! eval uat i ort^'or inferences! so that one 
might accurately record, evaluate, describe or justify those processes to 
one's self or to others, or so as to remedy perceived deficiencies in the 
general way one executes those processes. 

5.3 STATING ARGUMENTS: To present arguments which communicate one's 
grounds for accepting some claim! their logical force in supporting that 
claim, and, as necessary, meeting objections to the premises one relied on or 
the reasoning one employed. 

6: MONITORING: To sel f -*consc i ousl y regulate one's coqnitive activities, the 
elements used in those activities, and the results produced by those 
activities, particularly by applying analyzing and evaluating to one's/bwn 
inferring with a view toward confirming, validating, correcting or questioning 
either one's reasoning or one's results. 

6.1 SELF-EXAMINATION : To reflect carefully on one's own reasoning and 
verify both the results produced and the correct application and e)(ecution of 
the cognitive skills involved; to make a thoughtful meta-cogni t i ve self- 
assessment; to reflect on the extent to which one's thinking is influenced by 
deficiencies in one's knowledge, or by stereotypes, prejudices, emotions or 
any other factors which constrain one's objectivity or rationality; to reflect 
on one's motivations, values, attitudes and interests with a view toward 
determining that one has endeavored to be unbiased, fair-ninded, thorough, 
objective, respectful of the truth, reasonable, and rational in 'TOffiing to 
one's interpretations, analyses, evaluations, inferences, or expressions. 

6.2 SELF-CORRECTION: Where self-examination reveals errors or 
deficiencies, to design reasonable procedures to remedy or correci:, if 
possible, those mistakes and their causes. 

f #*# f t ftf 

R OOND 5-A> PART 2jl ZZ ENDORSEMENTS, 

1) Do you endorse the above statement as useful for purposes of 
asse.^'sing the skills dimension of CT at the lower division college 
level? 

2) Would you be willing to have your name listed in association 
with the above description of CT skills as a contributing member 
of the Delphi research project which generated it? 
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ROUND 5-A 



tJlILL I 11 Rg«Kt i on s 



The responses to ROUND 4 included sone telling observations, thoughtful 
objections and uel 1 focused criticises. Sharing these Hith others in 
the Delphi group is essential to the Delphi process. Pve included as 
fliany as is feasible and a?i can be understood outside the contoxt of 
whatever ROUND 4 item may have pronpted then. Rather than use of your 
time reading positive co/n/nents, of which there were many, Tve stuck 
chiefly to the critical ones. 

INSTRUCTIONSi Read and consider what our colleagues are telling us. In 
addition to the adjustments already incorporated into the above draft 
statements regarding CT and CT assessoent, what other responses and 
ifflpr ovenients should we nake? In the light of these comments and other 
concerns that cone to nind, what specific recommendations should ne 
include in our report to the fiPA Coooittee on Pre-College Philosophy? 



* "A test cannot be considered in the abstract, without Horking 
out its intended use and intended users, the specific population to be 
tested, and the d i scr i tni nat i ons the test would be required to rake«.« 
The domain the test covers is governed by this context, and concepts 
that constitute sc*b::lassifications of the domain are arranged in 
different ways fron the ways in which they might be arranged for a test 
Hith the same name but a different purpose... CT can be subcl assi f i ed 
in a number of legitimate nays, with laany of the same elements 
recurring in different places in different classification schemes..." 

* "There's hardly anything we now need more than improved CT 
tests. I've used ... in pre-post testing for several years. It's the 
best I know of. CBut. . . ] 

^ "...to make sense of CT we nust aake sense of the critical 
thinker. The cognitive must be discussed in relation to the affective, 
and both must be discussed in relation to their roles in the real 
world. Curriculum and assessment rust be put into some broad context. 

There is no one right definition of CT, and in testing we should 
never confuse testing for oicro-skills with testing for CT itself. 
Most CT tests are micro-skill tests only. They are valuable, but only 
in a qualified way. The Delphi project seems well on its way toward 
confusing the part with the whole." 

* "The [Round 4 list of skills] seeas fine to me. Ihere are 
definite li/iits to armchair analysis, and untii saeone actual ly starts 
trying to Measur e these things , it i $ difficult to itnow just bou to 
revise the list of CT skills and sub-skills." 

* "Please interpret rjy responses cautiously. I have not spent 
ffiuch time trying to define CT. My business is trying to ido3 
assessment once you 9uy5 arrive at clear and useful defir.itions/'' 

» "We cust focus on the purpose ar\d target audi c nee of an; given 
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CT test. Only that way can we fine tune our definition of CT, or 
should 1 say our "theoretical construct" of CT." 



♦ "Rather than limit our conception of CT to achieve a certain 
kino of test) why not sifnply fftake tnore (nodest claims about the test? 
Why not say that you are testing tisp^cis of CT? This betfi^b nore 
justified. Otherwi:se thu test becones the tail that wags the dog... Be 
honest about what viv are and are not testing by multiple choice tests. 
Lfit's not reduce rich and complex realities like CT to that which can 
be directly tested in the multiple-choice format." 

* CSome of your descriptions] are ditficult to tesit without open- 
ended items... [For example] it's difficult to test vormulation of 
strategies, HC exams focus nore on recognition of best ntrutegy. 

t "The categorization systen. . . gi ves little guidance on what 
precisely to teach and test. For instance, under "evaluating" is "to 
assess the credibility of statements. But what should be taught when 
doing this, and what should be tested when trying to find out what 
skills students have? If teachers are supposed to act based on what we 
produce, I believe we need to include criteria for assessing 
credibility. Again I refer you to...** 

» "CT is deeply connected in those individuals who are successful 
critical thinkers to a pervasive self consciousness about one's own 
thinking and reasoning processes. Such self consciousness should be 
deliberately cultivated in our students and should therefore be 
included somewhere in the overall description it is an intrinsic 
part of the CCT] process. It flonit' s the selection, application and 
interlinkage of the various relevant processes." 



SELECTED fiE$POH$lS OJi Ci AHJi DEFIHIHO C± 



» "Until we have a theory of reasoning <a combined normative 
theory of informal logic along with a descriptive theory of cognitive 
processes) we will not be able to spell out CT skills non-arbitrary 
with completeness and precision.... He are produc j ng. . « a fraeework, 
[that is] a list of concepts used for understanding a domain. If that 
is so, it will play hell out of our attempts to assess CT skills. I 
doubt that we will be able to manage construct validity for any 
conventional HC test with our Clist of CT skills]. Yet, what we are 
coming up with is extreraely valuable if we focus on performence 
assessment. Our franework gives us a passably good set of criteria by 
which one would judge good perforrsance on CT tasks. The criteria are 
developed by experts us and we are currently judging whether we 
accept them or reject them, another step in the process of developing a 
goo'i performance assessment. The next step would be to distribute 
typical essays, (good, bad and ugly ones) and aim at some consensus in 
telling the good ones from the bad ones." 

t "Me should resist the assertion that CT is donain dependent." 

« "I'm beginning to think that to have done all that Cwe have 
done] and not have tried to define CT may turn out to be a aistake... 
What makes (the list of] i/iportant, indeed basic, intellectual skills 
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Cin Round 4 J a list o)f critical thinking skills? It could just iis 
well be a lisi'. of rational thint:ing skiUsi or logical thinking skills 
or higher order cognitive skills. ...1 fail to see how this list 
captures the force of the word "critical". tEtyiDological I y] the 
meaning of "criticai'* is judging, evaluating, estimating the worth of 
so/nethi ng. . * A critical thinker is someone whu renders an opinion on an 
intellectual product.,.by assessing the strengths and weaknesses of 
that product,.. Doing so requires the capacity to elicit and apply 
standards, principles and criteria. None of the CRound 4] list is 
really this skill. If we asked for a list of problem solving skills, 
would we get the same list Cas in Round 43? If then either there 
is no conceptual difference between the two, which I think is wrong, or 
else the list fails to capture what is distinctive about CT skills. CT 
also connotes "crucial". CHere] the CRound 43 list 'fares better, 
because. ..these six skills are crucial --i.e., essential for 
intellectual survival. The problen is that the list is so bread and 
wide-ranging that it is not clear what intellectual skills have been 
excl uded. " 

* "If one identifies CT as that which makes a critucal person to 
be what he is^ then CyourJ narrow concept of CT is inadequate. CT 
is... an answer to the general probleo of conformity, prejudice, 
narrow-mindedness, end irrationality in the world. CT is what one does 
to achieve autonomy and independence of thought, to lesson one's 
prejudices, to broaden one's perspective, and to becone tiiore rational.'* 

♦ "The ftain overall worry is that the categories are Much too 
broad. When we get down to testing it will not be for sonething like 
"assessing logical strength" or "clarifying" but very specific skills 
such as "recognizing whether sunething is a necessary or sufficient 
condition; recognizing the difference between if p, then q's and if q 
the p's etc. The lines of demarcation are very unclear e.g. between 
psychological and logical (epi stemological ) criteria concerning say 
observat i on. " 

« "I have no major quarrel with any parts of your organization, 
[but] your emphasis leaves out a major component of CT the 
dispositional component and the set of values inherent in being a 
critical thinker... I think it is a good working model of CT skills, 
but an incomplete picture of being a critical thinker." 

* "There is no attention. »• to the dispositions... characteristic 
of CT. These... are as essential to CT as are the knowledge and 
information used in the processes." 

t "The dispositions are at least as important as the skills. I 
suggest their addition." 

t "CT works by recognizing and criticizing sources of inforeation, 
by drawing implications fron given naterials, identifying assuiptions, 
noticing relationships of consistency, inconsistency, inplications and 
contradiction, inferring Interesting consequences, recognizing, 
analyzing and evaluating arguments and constructing them as well. Of 
course, there's a lot niore to it." 
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» "Arguing is not listed as a separate skill!" 

» "No fnention is made oi understanding another's purpose... this 
would fall under interpreting, I suppose." 

» "I disagree with the list of CT skills as described. 
Interpreting is obviously a cognitive operation. But it is not a CT 
operation. Expressing is essentially low level communication, not 
gtjnerative in the sense of CT. Monitoring is -eieta-cognitive. The 
problem is that this description is so broad "critical" thinking gets 
lost in all the other kinds of thinking. This blurs the nature of CT 
beyond '•ecogni tion. . . CT is "judging the worth, accuracy or 
significance of something." 

* "There are several items I mess overall. They may be subsumed in 
some of the processes you have listed, but very few people will be 
conscious of them unless they are brought out explicitly. One is the 
capacity for arithmetical reasoning with ratio and division it 
begins with word problems in Sth and 6th grade arithmetic and carries 
up to exactly similar reasoning with concepts such as density^ 
composition, contraction! rates of change, in more sophisticated 
settings. It includes the ration reasoning that goes with scaling 
areas, volumes ratesi etc. This capacity is profoundly important in 
any CT that involves nuraerical information (whether it be scientific, 
economic, sociological, psychological... I'm talking about arithietic 
and not u^the^at ics at the level of calculus or even algebra. CA 
second capacity to include is] "correlational reasoning." Finally I 
diss e»:plicit inclusion of the process of translating symbols (e.g. 
graphs, numerical data, histograms) into words or words into 
corresponding symbols. Such translation is essential to much CT.'' 

« "CThe concept of CT should also include] di scri ai nati ng 
explicitly between the factual or experiential input and the inferences 
drawn, identifying gaps in available information and identifying 
irrelevant or superfluous information, and it should include the 
capacity to consider some situation in the abstract and, by applying 
relevant governing principles or constraints, arrive at reasonable and 
plausible conclusions about the outcomes that would result from the 
imposition of some change — hypotheti co-deducti ve reasoning." 

t "Include something about cppos 1 1 ioual reasoning taking the 
part of one with whom one disagrees. Also include hypothetical 
reastning reasoning from suppositions and hypotheses. 

* "Taxor.omy" is not a good word for what we've got; it's nore like 
a list. KbVing a taxonomy in biology and education is to have a 
hierarchical set cf categories such that each subsequent step in the 
hierarchy subsumes th(? steps below it. Me don't have that here. 

« "[Conjecturing] is creative thinking not CT. 

t "ITo have included seeking and gathering evidence] is ambiguous. 
Me are not sci enti sts. 

* ''[Regarding expressing,] I don't see any strong rationale for 
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extending the focus to caniniuni cat i ng the results of CT, 



* "None of Cyour si?:] listed items are skills let alone sub- 
skills. They are general categories into which many (at least dozens 
of) distinguishable skills may be luniped. This iS important. 
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California State University, Fullerton 

Fullerton. California 92634 



Department of Philosophy 
(7 i4) 773-3611 



DELPHI ROUND 5-B 



Mar. 6, 1989 



Dear Delphi Colleagues, 

This letter -fallows the outline in the Round 5-A letter — 
questions -first, background in-formation second. The focus here is on 
■fleshing out our conceptualization o-f CT beyond the revised list a-f CT 
abilities presented in Round 5-A. Speci-fically this letter responds to 
your comments regarding those dispositional and normative dimensions 
many include when describing CT. The implications for K-12 and college 
level assessment, curriculum development and pedagogy of including 
either of these dimensions are crucial for our effort. You're asked for 
your ideas about this, too. 

To get things started, on the next page you'll find a draft 
statement regarding the dispositional dimension of CT. Following that 
is a draft stal-ement regarding the? normative dimension. These two draft 
statements are based on your comments regarding ne«=?ding a fuller 
conceptualization of CT and an analysis of commonly referenced concepts 
of CT which appear in the literature. Have at those draft statements. 
Amend, edit, comment, accept, reject — whatever you think will help us 
be able to present an intellectually credible and educationally useful 
conceptualization of CT. 

Coming soon: Round 5-C focuses on our report to the APA Pre-CoUege 
Philosophy Committee. It will include a proposed outline of that; 
report, showing how the various pieces of the assessment puzzle we've 
worked on for over a year now will be incorporated. It also picks up 
the two pieces from Round 4 which haven't been addressed yet, namely the 
input-output model Cwhich we rather roundly rejectedD, and the statement 
of what a cognitive skill is and how cognitive skills might be assessed. 

I'll be speaking on March 24 at 1:00 p.m. at the Pacific Division 
meetings of the APA in Oakland CA, sharing a little of what we've been 
doing and the direction things seem to be taking. If you happen to be 
in the neigtiborhood, please stop by so we can visit. 

Knowing you are very busy, I beg mercy and ask you to reply as soon 
as is reasonably passible. A quicl-cie note with general comments is 
helpful, if you won't have time to go through things in careful detail. 

Thanks for your continuing support and involvement- 



Yours sincerely. 



ERIC 
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Pari: 1: The Dispositional Dimension of CT 
INSTRUCTIONS Reflect on the following statement. Edit, amend, or 
revise as yoa see -^it. Some background information is presented in the 
second half of this letter- Please respond to the following questions: 

1) Should our final report include a statement on the dispositional 
dimension of CT? Why, why not? 

2) What implications or recommendations for K-12 and college lower 
division level assessment, curriculum deveflopment and pedagogy follow 
from including a dispositional element in our conceptualization of CT? 

CT The 0 i spos it i oTial Oi^etiSion 

Ifi ^dditicfi to its cogiiitive skills diiertsiou , CT bIso iri\^olves certaiTi 
cogriiti\^e dii^positionSf persohal traits or intel lectusl \/irtues ^hich are 
crucial to its broad and successfu 1 use. Just as wit/j the cogriiti\^e 
abilities di^ensiort oi CT , w/ien corice i v ing of CT assess^eui or an 
instructional pr ogr ai^ in CT ^ it is i^ par t ant to cons ider ways of developing 
MBterialiif teaching strategies and assessment tools p^hich focus on CT 
dispositions. The cult ivat ion of CT cognitive dispositions is particularly 
inportant in insuring the use of CT abilities outside the narro>^ CT 
instruct ional setting. Persons Nho have developed the intellectual virtues 
and personal traits listed beloN are auc/) lore likely to apply their CT 
skills than are those Nho knot^ the skills but lack the intellectual 
discipline to use the^. 

The critical thinker is one w/)o possesses and cultivates the cognitive 
d i spos it ions , per son a I traits or intellectual virtues of 

* seeking a clear statement of issues, questions or problems, 

* curiosity in exploring issues and seeking information, 

* eagerness in seeking and in enp ley ing CT abi 1 it ies , 

* openness to seek and to cons ider divergent v ieNS or alternatives, 
^ trust in the processes of reasoned inquiry , 

* per sever ance , diligence and discipline in keeping ^ell-in forced, 

* honesty and hu9ility in appraising one's own reasoning and vieNS, 

* >ii 1 1 ingness to seek , use and Mention credible sources , 

^ prudence in suspending or waking judgments and in taking or chang ing 
one's pos it ion , 

* NisdoM and persistence in the use of appropriate, defensible and 
r e levant cr iter ia, 

* e t or t to addr ess issue s reasonably and to retain r e levant to the 
basic concern or prob len . 

* precision t to the exten t the subject permits, 

* orderliness in the treatnent of complex issues and processes. 
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Part 2: The Normative Dimension of CT 
INSTRUCTIONS Reflect on the following statement. Edit, amend, or 
revise as you see iit. Some background information is presented in th 
second half of this letter. Please respond to the follovjing questions: 

1) Should our final report include a statement on the normative 
dimension of CT? Why, why not? 

2) What implications or recommendc^tions for K-12 and college lowe^r 
division level assessment, curriculum development and pedagogy folioN 
from including a normative element in our conceptualization of CT7 

CT 7he Hornaiive DiMension 

If! addition to its coguiiive skills and disposi tional dimensions f CT 
also involves certain normative features ^hich govern its proper use. 
Under St anding that in making this stateM€nt we are going beyond a purely 
descriptive analysis, we judge it vital to include this normative component 
in our conceptual i zat ion of CT . In a free and rational society education 
must be tore than skills training and more than the inculcation of a set of 
personal disposit ions . Education must include the nurturing of those civic 
and personal values Nhich insure that the heritage of intellectual fair^ 
mindedness and political freedom wj'ii be passed to futurn gener ai i ons 0 

Properly used, CT contr ibut es to the fair-minded analysis anc resolut ion 
of issues , rather than the blind, soph :st ic 9 or irrational defense of v i ep^s 
kno^n to be intellectually defective or biased • 

Properly usedp CT unites those w/)o wouid reason together objectively in 
a ref lect ive and intellectually sound process, even if their final judgments 
and analyses should disagree^ 

Properly used, CT promotes rational autonomy, intellectual freedom, and 
the objective investigation of any issue or concern ^^hat soever . 

Properly used, CT treats all affected persons ^ith sensitivity and ^ith 
equal respect and dignity it is never exclusively self-interested, 
abusive, coercive, or without integrity and good faith. 

Thus, in a free, open and rational society, w/ien conceiving of CT 
assessment or an instruct ional program in CT , one ought to consider ways or* 
developing materials, teaching strategies and assessment tools Pihich insure 
the normative dimension of CT wlii be developed in students, in addition to 
CT skills and d i s pos i 1 1 on s . 
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Part 3 —BACKGROUND 



For your consideration, here are the views of six widely cited persons 
regarding what CT involves. 

1) CV is, nn. "Active persistent and careful consideration of a belief or 
supposed form of knowledge in light of the grounds that support it and the 
further conclusions to which it tendb" John Dewey. 1909 

2) CT "is reasonable reflective thinking that is focused on deciding what to 
believe or do." In addition to 12 CT abilities, CT also includes 14 
dispositions. Namely; "to seel: a clear statement of the thesis or question, 
to seel: reasons, to try to be well informed, to use credible sources and 
mention them, to take into account the total situation, try to remain relevant 
to the main point, to keep in mind the original or basic concern, to look for 
alternatives, to be open-minded, to take a position when the evidence and 
reasons are sufficient to do so, to !>eek as much precision as the subject 
permits, to deal in an orderly manner with the parts of a complex whole, to 
use one's CT abilities, to be sensitive ^o feelings, level of knowledge, and 
degree of sophistication of others" Robert Ennis, 1987. 

3) "CT is an attitude of being disposed to consider in a thoughtful way the 
problems and subjects that come within the range of one's experience; 
knowledi^e of the methods of logical inquiry and reasoning; and some skill in 
applying vhose methods.,." E,Q, "recognire problems, find ways to meet them, 
marshal pertinent data, recognize assuniptions use language well appraise 
evidence evaluate statements, see logical relationships, draw warranted 
inferences, test one's own conclusion, reconstruct one's beliefs based on 
experi.2nce, render accurate judgments." Edward Glaser, 1941, 

4) •*CT can be characterized as skeptical, radical, disillusioned, problem- 
seeking, holistic, judgment -oriented, non-algorithmic, constructive, 
comprehensive, empathic, meta-cogni ti ve , higher-order, reflective reason- 
governed , logical, self-correcting, context-sensitive, criterion -referenced, 
cr i si s-'or i ented , normative, inferential, pragmatic, purposive, relational, and 
open- minded." Matthew Lipman, 1988. 

5) "A given student, S, is a critical thinker in some area requiring mental 
effort, X, if S has the disposition and skill to solve problems in X using 
some subset of^the available evidence E pertinent to activities in area X, 
"CT" has an identifiable meaning but the criteria for its correct applica.tion 
vary from field to field," John McPeck, 1981. 

6) CT is disciplined, sel f -di r ected thinking which exemplifies the perfections 
of thinking appropriate to a particular mode or domain of thinking. It comes 
in two forms. If thinking is disciplined to serve the interests of a 
particular individual or group, to the exclusion of other relevant persons and 
groups,... it is sophistic or weak sense critical thinking. If the thinking 
is disciplined to take into account the interests of diverse persons or 
groups, it is fair-minded or strong sense critical thinking." Richard Paul, 
1988. 
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Here are your comments regarding statrnient Hib in the caveats and cautions 
section of Round 4, "Jf CT is a set of dtiitudes as weJJ as a set of 
skills, that poses no problei) for assessment because we can just develop 
Mays to assess the CT attitudes too,'' 

"Difficult to do!" 
"CThisl seems too facile. To be sure, rhetoricians and social 
scientists study attitudes and so have developed ways to ascertain or measure 
certain attitudes. But does this mean that all attitudes can be so tested? How 
can we talk about whether certain attitudes can be tested until we have 
enumerated those attitudes and so know specifically what we are trying to test? 
Also, can't some aspects of proficiency in CT, or beinn a critical thinker, 
involve both skill and attitudes combined? If CT involved recognizing one's own 
prejudices and stereotypes, one must have an attitude of willingness to confront 
one's stereotypes and the skill to delineate and recognize •them. Attitude and 
skin would go hand in hand. We don't just have a set of skills and as set of 
attitudes. This is too atomistic a description." 

"Agree" Cwith (6) as stated, no comment.] 6 people. 

"Agree, we should teach skills and dispositions, the attitudes are 

minor . " 

"Agree, but attitudes are harder to test." 

"Agree. CT is more a set of attitudes than a set of skills... It is 
easier to change attitudes in a single course than to improve skills 
substantially. Changing attitudes and fostering sel f -*monit.iiring should lead to 
lifelong improvement in a student's CT performance. So it's actually very 
important to say S'^niething about assessing CT attitudes..." 

"Disagree. The attitude/skill distinction isn't that sharp." 

"How?" 
" "Good luckf" 

"Unlikely." 

"There is a very great difference between skills and attitudes (although 
there are elisions at the boundaries of both concepts). Richard Paul has taught 
us most effectively [that] there is, or should be, a causal relation between 
skill and disposition such that the disposition might/should elicit, motivate, 
and work as side-constraints for the skills." 



Appendix C: Delphi Reaearch Lettere, PAGE 99 



Here. are your coiDinents regarding statement |}7 in the caveats and cautions 
section of Round 4, "CT in thinking skills. Saying CT is a sei of 
dttitudes »ay be a nay of describing nhai people nho are good at CT are 
like, but it is not a way of describing CT itself, 

"Good point ! " 

"CThis3 simply expresses a position dogmatically, and in the light of 
discussions by Richard Paul, Harvey Siegel, and others this is just not 
acceptable. To dismiss these positions without argument is just not 
philosophical!... CThisl simply rules out a major position on what CT is. 

"Agree" [with (7) as stated -- no comment]. 3 people. 

"The attributes required for CT can be ins.'iUed given enough time and 
the right emphasis of subject matter in CT, but not all CT subject matter 
instills proper attitudes. 

"Disagree" [with <7) as stated no comment]. 3 people. 

— "Disagree, Cwe] need to include some of the attributes." 
"Good move. Go for the CCT] itself!" 

"Agree, but I often find that performance does not change until I get a 
change in attitude. That the attitude has changed is best seen in the improved 
performance... The most important attitude is "1 really want to find out what 
you believe, and what your reasons for believing it are. Then when I am sure 
that you agree that I have understood you, I will think about and state where I 
differ f rom you. " 

— "Agree, but our ultimate concern is to encourage people to think 
critically when it is appropriate to do so. And that requires a critical spirit 
(critical attitudes) as well as thinking skills and knowledge," 

"Disagree. The attitude/skill distinction isn't that sharp." 

"Of course, some descriptions of dispositions are ways of describing 
what people who are good at CT are like, but I fail to see how it follows from 
this that they are not also ways of describing itself," 

t 
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California State University. Fullerton 

Fullerton. California 92634 ^' '^""^^O" 



Department of Philosophy 
<714) 773-3611 



DELPHI ROUND 5-C 

Mar- 10, 1989 

Dear Delphi Colleagues, 

This letter asks you to consider three things.-! 

<1> An outline of our report to the APA, (page 2), 

<2) A comment on skills and skill assessment, <page 3), 

<3> Specific recommendations we should make, (page 4) 

Your ROUND 4 comments on what a skill is and how a skill can be 
taught and assessed are very revealing. Take a look, for example, at 
the differences of opinion over the relative superiority of two of the 
strategies of assessment. The research Steve Norris is doing, whether 
or not Idiot savants can be said to be skilled, and the basic conflict 
between practical efficiency and accuracy in assessment, all found there 
way into your responses. (See pages 5 and 6.) This is another example 
of when an apparently off the wall question stimulated some useful 
ideas. 

Early responses to ROUND 5-A are coming in already. That's great. 

Please respond to the three round five letters as soon as is 
reasonably passible for you. I plan to start putting our report 
together this semester. Naturally you'll have a shot at it before 
final revisions. If all goes well we'll be ready to submit our report 
to the Pre-College Committee in the fall. 

Depending on the quality of the recommendations that come, we 
could be very close to wrapping things up. Thanks for your continuing 
support and active involvement. 

Yours sincerely, 

^^2~ — . 

Pete Facione 
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Part ii Proposed Outline o-F Our Report 

INSTRUCTIONS ; Consider the following outline of our Delphi 
research report to the American Philosophical Association 
Committee on Pre-College Philosophy. Comment, make any and all 
additions, deletions, amendments, changes you think reasonoble. 

CT — A Theoretical Construct for Purposes of Assessment 
Submitted to the Pre-College Philosophy Committee 
American Philosophical Association 

EXECUTIVE SUMMARY 

A. Delphi Conceptualisation of CT 

B. List of Delphi Recommendations 

C. Endorsement of Members of Delphi Research Panel 

REPORT 

I. Introduction 

A. The Concern for CT Assessment 

1. National interest and large scale assessment 

2. CT assessment in the individual classroom 
2. APA me-moers concern, college and K-12 level 

B. Formation of the Delphi Research Project 

1. Charge to the project director 

2. Description o-f Delphi Research Methodology 

3. Building the Delphi Panel of Experts. 

II. Delphi Findings, Points of Agreement and Disagreement 

A. Preliminary assumptions, CRounds 1 and 23 

B. Conceptualization of CT, CRounds 3, 4, and 53 

1. CT — the skills dimension 

2. CT — the dispositional dimension 

3. CT — the normative dimension 

C. General Comment on Assessing a Skill. CRounds 4 and S-C3 

III. Recommendations 

A. General Considerations Regarding Educational Assessment 

1. Validity, Reliability 

2. Difficulty and Discriminability 

3. Purposes of a CT assessment 

4. Characteristics of persons being assessed. 

B. Strategies for Classroom CT assessment 

1. Some questions to ask one's self 

2. Assessment, pedagogy and curriculum development. 

3. Suggestions on putting together a classroom CT test. 

C. Comments on Large Scale CT assessment 

IV. Appendices 

A. A Quick List of Purported CT Assessment Tools 

B. A CT Bibliography with emphasis on Assessment 

C. Delphi letters from each Round 

D. Response rates to each Delphi and other tabular data 

E. List of Delphi Panel of Experts 
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Part 2: General Comment on rtssessina A Skill 



INSTRUCTIONS; This revision is based on you Round 4 comments, Piease 
edit. You migh'' first read your colleagues' comments, on pp* 5 and 6. 

A skill is the ability to perforji certain processes or procedar es oore or 
less weiJ^ Haying a skill includes being able to do the right thing at the 
right t/ae. So, being skilled at something involves kno^^ing, perhaps 
implicitly or without the ability io articulate this knot>:ledge, a set of 
pr ocedures and ^hen to apply those procedures. It also involves having 5ooe 
degree of proficiency in executing those procedure's Reflecting on and 
improving one's own skills involves judging Nhvn one is or is not perforj^ing 
weiJ; and consider ing Nays of improving one's performance . 

It is generally thought that skills, par t icalar ly cognitive skills^ can 
be taught in a variety of Mays including naking the pr ocedur e s involved 
explicit, describing hoN they are to be applied and executed, explaining and 
Modeling their correct use, and justifying their appl icat ion • Teaching 
cognitive skills also involves exposing learners to situations where there are 
good reasons to exercise the desired procedures, judging their performance and 
providing the learners ^ith construct ive feedback regarding their proficiency 
and ways of improving it^ Instruct ion wight start ^ith s ituat ion s that are 
artificially simple, but it should culminate in situations that are realistic 
and complex. Teaching is not everything . ' The learners must cont r ibute a 
measure of effort, at tent ion , practice, desire, and self ^monitor ing. Teaching 
skills involves motivating learners to achieve higher levels of pro/jciency 
and; particularly in the case of CT , independence • It also may involve 
coaching learners on ho^ they can achieve those goals* 

Persons can be judged as being more or less proficient in a given skill. 
The first way of assessing is to observe the person performing the skill and 
make a judgment regarding the degree to Nhich the person possesses the general 
skill in question, A second way is to compare the outcomes (if any) that 
result from executing a given skill against some set of criteria. A third way 
is to query persons and receive the ir descriptions of the procedures and 
judgments they are using as they perform that skill, wouid use if they were to 
perform that skill, or did use when they performed that skill. A fourth way 
is to compare the outcomes (if any) that result from performing another task 
against some set of criteria, where the performance of that other task has 
been sho^n to correlate strongly Nith performance of the skill of interest. 
Ho matter which way is used, it is import.ant to ensure that the test 
condit ions foster an attitude in which the test-^taker s are disposed to use 
their skills as weJJ as they can; and are not cortstr a ined or inhibited from 
doing so. It is highly advantageous to cross check the results of any one way 
of assessment against the results of other ways. 
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Part 3: Recommendations 



INSTRUCTIONS ; Lonsidering what we know and don't know about CT 
assessment today and considering the direction oar Delphi effort has 
taken us, what advise, suggestions, recommendations should we make 
regarding CT assessment in the classroom or larger scale enterprises, 
(e.g. naticnal, state, district wide, college wide CT assessment)? 
Mention your intended audience, and be au< precise about the 
recommendations as possible. Le.g. To ETS ai\d ACT we should say, 
to teachers o-f primary grades we should say,...; tu professors who teach 
CT at the college level we should say,...; to the resear rh community in 
cognitive psychology we should say,...Il. 

To help you think through this most important matter of making 
recommendations, I've drafted some questions. You need not respond to 
them. They're only to stimulate thinking. Rather, formulate specific 
recommendations. Oh! A plea for mercy: I have no staff, so please don't 
ask that I search for things you might have once said about this 
someplace else.) 

K What questions have you found it useful to ask yourself regarding your 
own assessing of CT in teaching or professional work? 

2\ How night a college age person who possc-ssed the CT abilities we have 
described be able to show that she had those abilities? 

3. What about showing that she has the designated CT dispositions or 
normative attitudes, (should we decide to include those in our final report)? 

4. HoH night a person in elementary school, junior high or high school 
show that she has those CT abilities, dispositions or normati.ve attitudes? 

5. What questions or tasks might you ask a group of persons to undertake 
if you wanted good evidence regarding which persons were better at CT than 
which others? 

6» If you were doing a workshop for teachers at some grade level, what 
would you tell ther about CT assessment? 

7. If you were serving as a consultant to some organization which wanted 
to initiate large scale CT assessment, like a school district, university, or 
state department of education, what would your recommendations be? 

8. If a colleague and friend asi;ed what you really thought about how CT 
can best be taught, learned and (neasurf.^d, what would you tell your friend to 
do or avoid doing? 

9. What makes you optiaistic about CT assessment? 
10. What do you most fear about CT assessment? 
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Here are some of your comments on the idea o-f a skill and how 
skills can be taught and assessed. 

* Your ways of teaching skills are conspicuous in their lack of 
explicit emphasis on educational values including the use of morally justified 
procedures in teaching. Similarly your ways of learning lack such 
educationally important activities as explaining, understanding and 
justifying. " 

« "Mental skills are no less directly observable than physical skills. 
Skillb are abilities, and abilities can't be observed. Thus, obser'virig the 
skill as it is being perforned is impossible for any skill." 

« "I have ray doubts about ... the implicit mind/body dualism. 
Performances can be measured directly (e.g. by observing a surgeon's technique 
and results) and indirectly (e.g. by a paper and prncil test of knowledge of 
anatony). Both constitute a sample from which inferences are made to the more 
general skill, and this is all we need to concern ourselves with..." 

« ^'Your description exclude any effort, visualization, mental rehearsal, 
or using of any inner resources a person brings to skill learning and skill 
performance — concentration, automati ci ty , goal directedness etc." 

« "CThe narrative] presupposes that if one has a skill, one consciously 
knows the intricacies and interworking of that skill. Often this is simply 
not the case. ... Indeed idiot savants nay be said to possess skills with 
little or no understanding of how they perform skilled activities. ... (Those 
who] understand Chow they arrive at the correct result] are better critical 
thinkers [than those arrive at the result] with no real understanding of the 
rationale. 

« "...Don't we observe performances which we interpret as being skillful 
or being evidence of skill at a certain degree or level." 

* "It seems odd to talk about performing a skill, given the initial 
equating of a skill with the ability to do something well." 



»** CThe second way i^ superior to the third...] 

"because the third way can be employed only after it has been 
settled that the person has the skill, and that can be settled only by 
comparing outcomes against criteria, i.e. by employing the second way. In 
short the third way is parasitic on the second. Beside, a skilled person may 
not even be able to describe the procedures and judgments used when executing 
the skill." 

''for the assessment of average people, because (1) CT skills are 
generally employed unconsciously so the third way just won't work for average 
people, and (2) it's fast, cheap and yields unambiguous (if not wholly 
trustworthy)' answers... But the third way is superior when dealing with, 
experts, whom we can expect to be conscious of their procedures. If we can 
get an account of their actual thought processes, then we can assess those 
processes directly rather than indirectly." 

"if instead of outcoie we also include how the answer is arrived at; 
thus an idiot savant can produce answers to complex equations but not by any 
process we understand (or she does); if we lioit ourselves to outcomes in 
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judging skilly we'll fni'Ss the di5tir>ction between the highly skilled 
arithmetic operator an the idiot savant. Since I would want to claim that the 
idiot savant is not a case of skill at all, this presents sone difficulties." 

"because performance is what we want^ provided that outcones include 
the steps taken. That is, if a student makes a good judgment on someone's 
reasoning, I would want to know how he/she got to that judgment." 

"because it is much easier to carry out in practice, provided that 
there are enough items to compensate for accidental correct and incorrect 
outcomes. • • " 

"because it may be less ti me--consuming, and because it would be 
easier to design instruments for..." 

"because it does not depend completely on the testimony of the 
individual about what occurred. That's a strength because the literature on 
self-reports suggests enormous difficulties with the agent's own accounts of 
their activities. If we have the product, and the account of the process 
which yielded it, we're in better position to judge the degree of skill." 

[The second way is not superior to the third. ..3 
"because the second is a prerequisite for carrying out the third; 
that is, to do a credible job of querying a person must fist have completed a 
strong comparison and examination of the argument" 

^ "because, as far as CT is concerned, [the second] does not reveal 
the understandings and reasoning behind the answers given, but just assesses 
overt answers against some criterion. Clearly one could come to "wrong" 
answers depending on his understandings of the situatioQi and yet this 
reasoning not be a violation of, or deficiency in, CT. Suppose someone asks 
how many piles of beans we may divide 27 beans into, if we put S beans in each 
pile. Suppose someone says 6, because he allows that we can have small piles, 
the remainder can constitute a pile in its own right. Given his 
understanding, he has given the "right" answer. Does having his understanding 
mean he is not thinking critically? ...Obviously, the third way has its 
drawbacks, since it is far more consuming than checking outcomes against a set 
of criti^ria. A truly superior method of testing CT would incorporate both." 
"because there are often other possible explanations of the 

outcomes." 

"because the outcome could have been accidental or a consequence of 
good luck, or bad luck, or in the Case of testing, copied." 

"because the third method might reveal that a person has good CT 
skills where the second method suggested poor skills." 

"Cif one cannot specify] the criteria against which to judge the 
process-outcome . " 

* "I do not regard the second and third ways Cof assessing] as 
constituting a hierarchy with one way superior to the other. I use both ways 
in assessing students and check the two against each other, whenever I can. 

« "Part of the problem with analyzing CT into skills is that a good deal 
of the knowledge that makes up CT (quite apart from the so-called knowledge 
which it presupposes) i^ prepositional knowledge and not a skill. One cannot 
spot an argument or assess it validly without being able to define what an 
argument is and what valid is. Forcing CT into skills is basically confused." 

no 
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California State University, Fulierton 

Fullerton, California 92634 



lERIC 



Department Of Philosophy ^'^ ^^^^ 

Dear Delphi Colleagues, 

Dozens of responses to rounds 5-A, S'-B, and 5-C have come in. Again y 
I find myself admiring your perceptive analyses and sound suggestions. 
And I am not just saying that to make you fell good- Your responses 
range from the outstanding to the excellent- Often I find myself writing 
"good point" and "remember" in the margins of your letters- 
Some of you have not responded yet the ROUND FIVE letters. But the 
notes or phone calls from those too busy to jump in to round five were 
certainly appreciated. If you still have round five on you desk and are 
wondering whether or not to respond now that the semester is coming to an 
end the answer is please, by all means. We need your input not just to 
keep up the quantity of participants but also for your ideas and 
insights. There is still time to reply before I start putting together 
the draft of our final report, a project I will not undertake until 
August or September. For the summer I've taken on a huge teaching load 
to help three of my children and fuy wife continue their undergraduate and 
graduate educations. 

I hope to make two presentations at the Sonoma State conference in 
August- One is a workshop on CT assessment, the other is tentatively 
called "The Marlboro Man and Broad-Shouldered CT" The paper, "Assessing 
CT and Building Consensus^ on CT," delivered at the Pacific Division 
meetings of the APA went well. I'm doing a similar presentation for the 
Education Colloquium of UC Davis next week and again in Baltimore in 
Sept. A couple of junior colleges and a couple of state universities in 
California have asked me tu do staff development presentations rn CT 
assessment- A favorite is "Thirty Ways to Mess Up a CT Test," another 
is "What is an Elephant, or Defining CT for Practical Purposes." 

The above is a not too subtle way of saying I'm available if you are 
planning something on CT. 

CSU Fullerton has given me some time next academic year to pilot 
test a CT assessment instrument. The curriculum committee at our 
university defines CT based on CSU executive order 338. For those of you 
outside CA, that executive order gives a general specification of CT for 
purposes of requiring a "CT course" be included in the general education 
program of all twenty of state universities- That definition is not 
inherently inconsistent with our Delphi findings, but it is far less 
sophiiDticated. If my research project goes along as planned, then by 
this time next year I should know if the CT assessment tool we will have 
put together is capable of detecting improvements in students' CT which 
result from their taking a 3 unit required lower division CT course from 
the Philosophy Dept- This is a dangerous question! Ironically, a key 
assumption in our experimental design is that the Philosophy faculty 
teaching CT are doing an effective job. The experiment is to find out if 
the assessment tool is sensitive enough to detect the difference 
assume our CT instruction is making. 

Have a good summer. -'-j.a 

^ . SinciSrely ^.-^ 

ir> The CaUlcrnla State Untv«r«iiw y < 
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California State University, Fuiierton 

Fullerton, California 92634-4080 



Department of Philosophy FINAL liOUND *** 

(714) 773-3611 

DELPHI ROUND 6 

Sept. 25, 1939 

Dear Delphi Colleagues, 

Please review the enclosed final d ra ft.^ Your 
your comments, substantive or editorial, are 
vital. Be sure to check the consensus 
statements in the Tables and the final 
recommendations. TO BE HELPFUL YOUR COMMENTS 
AND SUGGESTIONS MUST liEACH ME BY OCT. 30. 



The final report to the APA Committee on Pre-College 
Philosophy is due in November. Its intended audience is 
educators, at c\ny level, interested in CT, To clarify, as 
principle invest i^^ator my role is to express the Delphi findin.fc{s 
objectively, wli'slher they be poinLs of consen.'^ucj , majority 
positions, or minority opinions. I am not one of the 46 '.export 
participan ta . I hope you find tlie draft a clear and fair-minded 
expresriion of the delicate confluence of your expert opinions. 

ROUNDS r.A, SB and 5C were extremely useful. Of the 4 6 
•-•xperts, 26. 23 and 22 rerjponded in the.se rounds. ROUND f3A 
confirmed the .strong consensus regarding the cof^nitive skill 
dim'MiJjinn of CT . ROUND 5B .--howed that 61% of t^-^ experts hold 
that CT includes affective dispoiiitions , but 30% maintain CT does 
not. However, over 80% would be willin/j to use the same list of 
affective dispo.'iitions in describing the paradigm critical 
thinker. Although everyone reoogni.'^ed the personal and civic 
value of CT, only 17% argued that "CT" has a normative meaning. 
ROUND 5C produced several thoughtful recommendations relatin.*? to 
CT teaching and assessment. Be s ure t o consider the se ca refully 
and expa n d on the m if you thin k it would be of general u.?e "to K~l? . 
or college educators. 

In Delplii researcli once an expert expresses an opinion, even 
a dissenting one, it becomes a factor in the mix and flow of all 
subsequent argument and thought. Because of this, and because in 
Delphi research it is reasonable to assume that silence from busy 
experts is a sign of general accord with the direction of inquiry, 
I will continue to operate on the principle that unless you 
object, you find tilings generally acceptable. 

My respect for your collective and individual wisdom and my 
gratitude for your participation are so deep that I cannot 
possibly express either as fully as I feel them. Thank you so 
veri' much for being part of this two year adventure. I hope the 
final report does justice to what y ou th ink. 

I am most appreciatively yours. 
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